[2025-01-21 05:12:14,951 I 16549 16549] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed [2025-01-21 05:12:14,951 I 16549 16549] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. [2025-01-21 05:12:14,956 I 16549 16549] (gcs_server) event.cc:493: Ray Event initialized for GCS [2025-01-21 05:12:14,956 I 16549 16549] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE [2025-01-21 05:12:14,956 I 16549 16549] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR [2025-01-21 05:12:14,956 I 16549 16549] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB [2025-01-21 05:12:14,956 I 16549 16549] (gcs_server) event.cc:324: Set ray event level to warning [2025-01-21 05:12:14,957 I 16549 16549] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:42: Loading job table data. [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:54: Loading node table data. [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:80: Loading actor table data. [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 621dd511d43f0600c7fa977093088315fa64471d60b691074ed19ce6 [2025-01-21 05:12:14,958 I 16549 16549] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 [2025-01-21 05:12:14,960 I 16549 16549] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 58031. [2025-01-21 05:12:15,188 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 0 - DrainNode request count: 0 - GetAllNodeInfo request count: 0 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 0 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:12:15,188 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 25 total (5 active) Queueing time: mean = 81.385 ms, max = 225.583 ms, min = 1.495 us, total = 2.035 s Execution time: mean = 9.182 ms, total = 229.551 ms Event stats: GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 25.067 ms, total = 225.599 ms, Queueing time: mean = 175.095 ms, max = 225.278 ms, min = 1.495 us, total = 1.576 s GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 5.923 us, total = 29.614 us, Queueing time: mean = 38.121 us, max = 40.516 us, min = 34.907 us, total = 190.607 us PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.929 us, total = 11.718 us, Queueing time: mean = 112.775 ms, max = 225.583 ms, min = 225.516 ms, total = 451.098 ms event_loop_lag_probe - 2 total (0 active), Execution time: mean = 5.589 us, total = 11.178 us, Queueing time: mean = 3.505 ms, max = 6.895 ms, min = 116.207 us, total = 7.011 ms RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 3.890 ms, total = 3.890 ms, Queueing time: mean = 455.845 us, max = 455.845 us, min = 455.845 us, total = 455.845 us GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 8.875 us, total = 8.875 us, Queueing time: mean = 2.227 us, max = 2.227 us, min = 2.227 us, total = 2.227 us NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-21 05:12:15,188 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 4 total (1 active) Queueing time: mean = 112.016 us, max = 395.058 us, min = 15.917 us, total = 448.065 us Execution time: mean = 95.525 us, total = 382.098 us Event stats: event_loop_lag_probe - 2 total (0 active), Execution time: mean = 185.397 us, total = 370.793 us, Queueing time: mean = 216.074 us, max = 395.058 us, min = 37.090 us, total = 432.148 us GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:12:15,188 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4 total (1 active) Queueing time: mean = 160.651 us, max = 600.706 us, min = 13.354 us, total = 642.603 us Execution time: mean = 97.041 us, total = 388.163 us Event stats: event_loop_lag_probe - 2 total (0 active), Execution time: mean = 188.650 us, total = 377.301 us, Queueing time: mean = 314.625 us, max = 600.706 us, min = 28.543 us, total = 629.249 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-21 05:12:15,188 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4 total (0 active) Queueing time: mean = 572.049 us, max = 1.999 ms, min = 27.668 us, total = 2.288 ms Execution time: mean = 46.486 us, total = 185.943 us Event stats: event_loop_lag_probe - 2 total (0 active), Execution time: mean = 92.421 us, total = 184.843 us, Queueing time: mean = 1.013 ms, max = 1.999 ms, min = 27.668 us, total = 2.026 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:12:16,208 I 16549 16549] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=3c97e402ac1f323212309d4e3e7c6ac4544bfd0bfbd0bef1969903c3 [2025-01-21 05:12:16,208 I 16549 16549] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=3c97e402ac1f323212309d4e3e7c6ac4544bfd0bfbd0bef1969903c3 [2025-01-21 05:12:16,208 I 16549 16549] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 3c97e402ac1f323212309d4e3e7c6ac4544bfd0bfbd0bef1969903c3 registered, will try to reschedule all the infeasible placement groups. [2025-01-21 05:12:16,213 I 16549 16637] (gcs_server) ray_syncer.cc:377: Get connection node_id=3c97e402ac1f323212309d4e3e7c6ac4544bfd0bfbd0bef1969903c3 [2025-01-21 05:12:17,143 I 16549 16549] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 16479 [2025-01-21 05:12:17,143 I 16549 16549] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 16479 [2025-01-21 05:12:24,966 W 16549 16572] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:61114: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster. [2025-01-21 05:13:15,188 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 12 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 14 - cumulative published bytes: 12653 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:13:15,189 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 1104 total (4 active) Queueing time: mean = 1.988 ms, max = 225.583 ms, min = 727.000 ns, total = 2.195 s Execution time: mean = 554.343 us, total = 611.995 ms Event stats: event_loop_lag_probe - 242 total (0 active), Execution time: mean = 20.928 us, total = 5.064 ms, Queueing time: mean = 36.735 us, max = 6.895 ms, min = 1.848 us, total = 8.890 ms GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.589 ms, total = 227.207 ms, Queueing time: mean = 11.049 ms, max = 225.278 ms, min = 1.495 us, total = 1.580 s GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 18.544 us, total = 1.335 ms, Queueing time: mean = 19.371 us, max = 137.899 us, min = 2.227 us, total = 1.395 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 37.316 us, total = 2.649 ms, Queueing time: mean = 1.074 ms, max = 22.982 ms, min = 9.997 us, total = 76.260 ms InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 1.541 ms, total = 109.397 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s RayletLoadPulled - 60 total (1 active), Execution time: mean = 209.166 us, total = 12.550 ms, Queueing time: mean = 77.649 us, max = 1.351 ms, min = 10.621 us, total = 4.659 ms NodeManagerService.grpc_client.GetResourceLoad - 58 total (0 active), Execution time: mean = 1.104 ms, total = 64.035 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 58 total (0 active), Execution time: mean = 66.312 us, total = 3.846 ms, Queueing time: mean = 76.028 us, max = 221.037 us, min = 15.122 us, total = 4.410 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 33.295 us, total = 1.298 ms, Queueing time: mean = 73.755 us, max = 130.284 us, min = 12.508 us, total = 2.876 ms InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 490.606 us, total = 19.134 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 25.659 us, total = 615.818 us, Queueing time: mean = 86.941 us, max = 176.946 us, min = 22.231 us, total = 2.087 ms NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 467.118 us, total = 11.211 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms ClusterResourceManager.ResetRemoteNodeView - 21 total (1 active), Execution time: mean = 8.060 us, total = 169.256 us, Queueing time: mean = 60.361 us, max = 87.101 us, min = 27.397 us, total = 1.268 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s HealthCheck - 18 total (0 active), Execution time: mean = 6.316 us, total = 113.681 us, Queueing time: mean = 70.100 us, max = 124.274 us, min = 19.050 us, total = 1.262 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 72.002 us, total = 864.027 us, Queueing time: mean = 89.673 us, max = 163.101 us, min = 22.663 us, total = 1.076 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 567.371 us, total = 6.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.239 ms, total = 7.433 ms, Queueing time: mean = 56.419 us, max = 123.501 us, min = 51.173 us, total = 338.511 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:13:15,189 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 2734 total (1 active) Queueing time: mean = 36.730 us, max = 395.058 us, min = 3.424 us, total = 100.420 ms Execution time: mean = 224.099 us, total = 612.686 ms Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 1239 total (0 active), Execution time: mean = 450.480 us, total = 558.144 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1239 total (0 active), Execution time: mean = 38.855 us, total = 48.142 ms, Queueing time: mean = 78.413 us, max = 238.511 us, min = 9.421 us, total = 97.153 ms event_loop_lag_probe - 242 total (0 active), Execution time: mean = 25.975 us, total = 6.286 ms, Queueing time: mean = 10.419 us, max = 395.058 us, min = 3.424 us, total = 2.521 ms GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 7.907 us, total = 102.795 us, Queueing time: mean = 56.078 us, max = 90.190 us, min = 32.755 us, total = 729.018 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:13:15,189 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 398 total (25 active) Queueing time: mean = 22.410 us, max = 600.706 us, min = 2.914 us, total = 8.919 ms Execution time: mean = 137.724 ms, total = 54.814 s Event stats: event_loop_lag_probe - 242 total (0 active), Execution time: mean = 24.083 us, total = 5.828 ms, Queueing time: mean = 10.846 us, max = 600.706 us, min = 2.914 us, total = 2.625 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 38 total (0 active), Execution time: mean = 27.223 us, total = 1.034 ms, Queueing time: mean = 83.917 us, max = 188.600 us, min = 8.714 us, total = 3.189 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 38 total (24 active), Execution time: mean = 1.441 s, total = 54.761 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 14 total (0 active), Execution time: mean = 127.847 us, total = 1.790 ms, Queueing time: mean = 79.616 us, max = 119.406 us, min = 25.062 us, total = 1.115 ms InternalPubSubGcsService.grpc_server.GcsPublish - 14 total (0 active), Execution time: mean = 590.386 us, total = 8.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:13:15,190 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 252 total (0 active) Queueing time: mean = 19.280 us, max = 1.999 ms, min = 693.000 ns, total = 4.859 ms Execution time: mean = 25.826 us, total = 6.508 ms Event stats: event_loop_lag_probe - 242 total (0 active), Execution time: mean = 24.949 us, total = 6.038 ms, Queueing time: mean = 18.237 us, max = 1.999 ms, min = 4.195 us, total = 4.413 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:14:15,190 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 24 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 27 - cumulative published bytes: 22502 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:14:15,191 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 1823 total (4 active) Queueing time: mean = 1.217 ms, max = 225.583 ms, min = 727.000 ns, total = 2.219 s Execution time: mean = 412.250 us, total = 751.531 ms Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 21.719 us, total = 10.468 ms, Queueing time: mean = 22.764 us, max = 6.895 ms, min = 1.848 us, total = 10.972 ms GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 952.731 us, total = 227.703 ms, Queueing time: mean = 6.625 ms, max = 225.278 ms, min = 1.495 us, total = 1.583 s RayletLoadPulled - 120 total (1 active), Execution time: mean = 211.414 us, total = 25.370 ms, Queueing time: mean = 69.498 us, max = 1.351 ms, min = 10.621 us, total = 8.340 ms NodeManagerService.grpc_client.GetResourceLoad - 118 total (0 active), Execution time: mean = 1.142 ms, total = 134.732 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 118 total (0 active), Execution time: mean = 64.732 us, total = 7.638 ms, Queueing time: mean = 82.017 us, max = 221.037 us, min = 15.122 us, total = 9.678 ms GcsInMemoryStore.Get - 84 total (0 active), Execution time: mean = 18.677 us, total = 1.569 ms, Queueing time: mean = 18.262 us, max = 137.899 us, min = 2.227 us, total = 1.534 ms InternalKVGcsService.grpc_server.InternalKVGet - 83 total (0 active), Execution time: mean = 1.383 ms, total = 114.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 83 total (0 active), Execution time: mean = 38.143 us, total = 3.166 ms, Queueing time: mean = 927.538 us, max = 22.982 ms, min = 9.997 us, total = 76.986 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 33.044 us, total = 2.082 ms, Queueing time: mean = 70.241 us, max = 164.061 us, min = 12.508 us, total = 4.425 ms InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 469.442 us, total = 29.575 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 24.816 us, total = 1.191 ms, Queueing time: mean = 96.041 us, max = 188.417 us, min = 22.231 us, total = 4.610 ms NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 470.461 us, total = 22.582 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 41 total (1 active), Execution time: mean = 8.247 us, total = 338.125 us, Queueing time: mean = 64.332 us, max = 106.861 us, min = 27.397 us, total = 2.638 ms HealthCheck - 38 total (0 active), Execution time: mean = 6.161 us, total = 234.107 us, Queueing time: mean = 70.842 us, max = 124.274 us, min = 16.262 us, total = 2.692 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 72.914 us, total = 1.750 ms, Queueing time: mean = 90.781 us, max = 163.101 us, min = 22.663 us, total = 2.179 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 562.352 us, total = 13.496 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.249 ms, total = 14.991 ms, Queueing time: mean = 64.373 us, max = 123.501 us, min = 50.054 us, total = 772.480 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 790.661 us, total = 1.581 ms, Queueing time: mean = 28.401 us, max = 56.802 us, min = 56.802 us, total = 56.802 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:14:15,191 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 5506 total (1 active) Queueing time: mean = 38.401 us, max = 395.058 us, min = 2.358 us, total = 211.436 ms Execution time: mean = 214.431 us, total = 1.181 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 2499 total (0 active), Execution time: mean = 438.262 us, total = 1.095 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2499 total (0 active), Execution time: mean = 29.459 us, total = 73.619 ms, Queueing time: mean = 82.268 us, max = 238.511 us, min = 6.767 us, total = 205.587 ms event_loop_lag_probe - 482 total (0 active), Execution time: mean = 24.068 us, total = 11.601 ms, Queueing time: mean = 9.172 us, max = 395.058 us, min = 2.358 us, total = 4.421 ms GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 8.338 us, total = 208.457 us, Queueing time: mean = 56.478 us, max = 90.190 us, min = 26.706 us, total = 1.412 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:14:15,191 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 690 total (25 active) Queueing time: mean = 19.327 us, max = 600.706 us, min = 2.914 us, total = 13.335 ms Execution time: mean = 166.533 ms, total = 114.908 s Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 23.808 us, total = 11.475 ms, Queueing time: mean = 9.730 us, max = 600.706 us, min = 2.914 us, total = 4.690 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 51 total (0 active), Execution time: mean = 30.099 us, total = 1.535 ms, Queueing time: mean = 86.330 us, max = 188.600 us, min = 8.714 us, total = 4.403 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 51 total (24 active), Execution time: mean = 2.252 s, total = 114.840 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 27 total (0 active), Execution time: mean = 113.982 us, total = 3.078 ms, Queueing time: mean = 83.396 us, max = 133.342 us, min = 21.208 us, total = 2.252 ms InternalPubSubGcsService.grpc_server.GcsPublish - 27 total (0 active), Execution time: mean = 562.971 us, total = 15.200 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:14:15,191 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 492 total (0 active) Queueing time: mean = 14.200 us, max = 1.999 ms, min = 693.000 ns, total = 6.987 ms Execution time: mean = 25.052 us, total = 12.326 ms Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 24.595 us, total = 11.855 ms, Queueing time: mean = 13.571 us, max = 1.999 ms, min = 4.195 us, total = 6.541 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:15:15,191 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 36 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 39 - cumulative published bytes: 32318 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:15:15,192 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 2542 total (4 active) Queueing time: mean = 881.558 us, max = 225.583 ms, min = 727.000 ns, total = 2.241 s Execution time: mean = 351.608 us, total = 893.786 ms Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 21.609 us, total = 15.602 ms, Queueing time: mean = 17.962 us, max = 6.895 ms, min = 1.848 us, total = 12.969 ms GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 681.002 us, total = 228.136 ms, Queueing time: mean = 4.738 ms, max = 225.278 ms, min = 1.495 us, total = 1.587 s RayletLoadPulled - 180 total (1 active), Execution time: mean = 214.213 us, total = 38.558 ms, Queueing time: mean = 67.790 us, max = 1.351 ms, min = 10.621 us, total = 12.202 ms NodeManagerService.grpc_client.GetResourceLoad - 178 total (0 active), Execution time: mean = 1.172 ms, total = 208.678 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 178 total (0 active), Execution time: mean = 65.369 us, total = 11.636 ms, Queueing time: mean = 77.551 us, max = 221.037 us, min = 14.808 us, total = 13.804 ms GcsInMemoryStore.Get - 96 total (0 active), Execution time: mean = 18.577 us, total = 1.783 ms, Queueing time: mean = 17.287 us, max = 137.899 us, min = 2.227 us, total = 1.660 ms InternalKVGcsService.grpc_server.InternalKVGet - 95 total (0 active), Execution time: mean = 1.268 ms, total = 120.460 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 95 total (0 active), Execution time: mean = 39.582 us, total = 3.760 ms, Queueing time: mean = 819.108 us, max = 22.982 ms, min = 9.997 us, total = 77.815 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 32.330 us, total = 2.813 ms, Queueing time: mean = 69.312 us, max = 232.806 us, min = 12.508 us, total = 6.030 ms InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 453.559 us, total = 39.460 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 23.385 us, total = 1.684 ms, Queueing time: mean = 90.953 us, max = 188.417 us, min = 17.215 us, total = 6.549 ms NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 461.680 us, total = 33.241 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 61 total (1 active), Execution time: mean = 8.474 us, total = 516.905 us, Queueing time: mean = 62.083 us, max = 132.714 us, min = 27.397 us, total = 3.787 ms HealthCheck - 58 total (0 active), Execution time: mean = 6.379 us, total = 369.957 us, Queueing time: mean = 68.816 us, max = 143.742 us, min = 16.262 us, total = 3.991 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 73.683 us, total = 2.653 ms, Queueing time: mean = 83.667 us, max = 171.832 us, min = 22.663 us, total = 3.012 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 555.187 us, total = 19.987 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.271 ms, total = 22.869 ms, Queueing time: mean = 60.651 us, max = 123.501 us, min = 30.234 us, total = 1.092 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 1.106 ms, total = 3.318 ms, Queueing time: mean = 41.409 us, max = 67.424 us, min = 56.802 us, total = 124.226 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:15:15,193 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 8278 total (1 active) Queueing time: mean = 37.386 us, max = 395.058 us, min = 2.358 us, total = 309.480 ms Execution time: mean = 211.145 us, total = 1.748 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 3759 total (0 active), Execution time: mean = 434.356 us, total = 1.633 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3759 total (0 active), Execution time: mean = 25.975 us, total = 97.639 ms, Queueing time: mean = 80.047 us, max = 238.511 us, min = 6.767 us, total = 300.897 ms event_loop_lag_probe - 722 total (0 active), Execution time: mean = 23.732 us, total = 17.135 ms, Queueing time: mean = 8.894 us, max = 395.058 us, min = 2.358 us, total = 6.421 ms GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 8.805 us, total = 325.773 us, Queueing time: mean = 57.994 us, max = 95.633 us, min = 16.403 us, total = 2.146 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:15:15,193 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 978 total (25 active) Queueing time: mean = 17.661 us, max = 600.706 us, min = 2.413 us, total = 17.273 ms Execution time: mean = 179.010 ms, total = 175.072 s Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 24.055 us, total = 17.368 ms, Queueing time: mean = 9.559 us, max = 600.706 us, min = 2.413 us, total = 6.902 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 63 total (0 active), Execution time: mean = 33.348 us, total = 2.101 ms, Queueing time: mean = 82.650 us, max = 188.914 us, min = 8.714 us, total = 5.207 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 63 total (24 active), Execution time: mean = 2.778 s, total = 174.990 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 39 total (0 active), Execution time: mean = 106.967 us, total = 4.172 ms, Queueing time: mean = 81.358 us, max = 152.481 us, min = 21.208 us, total = 3.173 ms InternalPubSubGcsService.grpc_server.GcsPublish - 39 total (0 active), Execution time: mean = 557.972 us, total = 21.761 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:15:15,193 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 732 total (0 active) Queueing time: mean = 12.775 us, max = 1.999 ms, min = 693.000 ns, total = 9.351 ms Execution time: mean = 25.580 us, total = 18.725 ms Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.283 us, total = 18.254 ms, Queueing time: mean = 12.335 us, max = 1.999 ms, min = 4.195 us, total = 8.906 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:16:15,193 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 48 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 52 - cumulative published bytes: 42167 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:16:15,194 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 3261 total (4 active) Queueing time: mean = 695.446 us, max = 225.583 ms, min = 727.000 ns, total = 2.268 s Execution time: mean = 327.568 us, total = 1.068 s Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 22.741 us, total = 21.877 ms, Queueing time: mean = 15.933 us, max = 6.895 ms, min = 1.848 us, total = 15.327 ms GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 530.462 us, total = 228.629 ms, Queueing time: mean = 3.693 ms, max = 225.278 ms, min = 1.495 us, total = 1.592 s RayletLoadPulled - 240 total (1 active), Execution time: mean = 230.605 us, total = 55.345 ms, Queueing time: mean = 69.592 us, max = 1.351 ms, min = 10.621 us, total = 16.702 ms NodeManagerService.grpc_client.GetResourceLoad - 238 total (0 active), Execution time: mean = 1.270 ms, total = 302.188 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 238 total (0 active), Execution time: mean = 68.731 us, total = 16.358 ms, Queueing time: mean = 81.392 us, max = 230.850 us, min = 14.808 us, total = 19.371 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 33.428 us, total = 3.710 ms, Queueing time: mean = 72.567 us, max = 239.072 us, min = 12.508 us, total = 8.055 ms InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 465.172 us, total = 51.634 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 108 total (0 active), Execution time: mean = 18.510 us, total = 1.999 ms, Queueing time: mean = 16.677 us, max = 137.899 us, min = 2.227 us, total = 1.801 ms InternalKVGcsService.grpc_server.InternalKVGet - 107 total (0 active), Execution time: mean = 1.185 ms, total = 126.808 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 107 total (0 active), Execution time: mean = 40.976 us, total = 4.384 ms, Queueing time: mean = 736.416 us, max = 22.982 ms, min = 9.997 us, total = 78.796 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 24.153 us, total = 2.319 ms, Queueing time: mean = 91.129 us, max = 188.417 us, min = 17.215 us, total = 8.748 ms NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 488.820 us, total = 46.927 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 81 total (1 active), Execution time: mean = 8.901 us, total = 720.982 us, Queueing time: mean = 64.438 us, max = 132.714 us, min = 27.397 us, total = 5.220 ms HealthCheck - 78 total (0 active), Execution time: mean = 6.896 us, total = 537.869 us, Queueing time: mean = 71.499 us, max = 173.947 us, min = 16.262 us, total = 5.577 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 78.489 us, total = 3.767 ms, Queueing time: mean = 88.825 us, max = 225.322 us, min = 22.663 us, total = 4.264 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 575.365 us, total = 27.618 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.259 ms, total = 30.224 ms, Queueing time: mean = 58.402 us, max = 123.501 us, min = 15.631 us, total = 1.402 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.222 ms, total = 4.890 ms, Queueing time: mean = 35.627 us, max = 67.424 us, min = 18.282 us, total = 142.508 us GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:16:15,194 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 11046 total (1 active) Queueing time: mean = 40.519 us, max = 395.058 us, min = 2.358 us, total = 447.571 ms Execution time: mean = 222.388 us, total = 2.456 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 5017 total (0 active), Execution time: mean = 458.765 us, total = 2.302 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 5017 total (0 active), Execution time: mean = 26.146 us, total = 131.173 ms, Queueing time: mean = 86.881 us, max = 311.554 us, min = 5.968 us, total = 435.883 ms event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.142 us, total = 23.224 ms, Queueing time: mean = 8.932 us, max = 395.058 us, min = 2.358 us, total = 8.593 ms GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 9.429 us, total = 462.007 us, Queueing time: mean = 62.835 us, max = 151.065 us, min = 16.403 us, total = 3.079 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:16:15,194 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1270 total (25 active) Queueing time: mean = 17.838 us, max = 600.706 us, min = 2.413 us, total = 22.655 ms Execution time: mean = 185.239 ms, total = 235.253 s Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.169 us, total = 24.213 ms, Queueing time: mean = 9.811 us, max = 600.706 us, min = 2.413 us, total = 9.438 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 76 total (0 active), Execution time: mean = 36.180 us, total = 2.750 ms, Queueing time: mean = 86.073 us, max = 188.914 us, min = 8.714 us, total = 6.542 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 76 total (24 active), Execution time: mean = 3.094 s, total = 235.154 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 52 total (0 active), Execution time: mean = 103.490 us, total = 5.381 ms, Queueing time: mean = 90.077 us, max = 190.637 us, min = 21.208 us, total = 4.684 ms InternalPubSubGcsService.grpc_server.GcsPublish - 52 total (0 active), Execution time: mean = 579.954 us, total = 30.158 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:16:15,194 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 972 total (0 active) Queueing time: mean = 12.725 us, max = 1.999 ms, min = 693.000 ns, total = 12.369 ms Execution time: mean = 27.242 us, total = 26.479 ms Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 27.036 us, total = 26.009 ms, Queueing time: mean = 12.395 us, max = 1.999 ms, min = 4.195 us, total = 11.924 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:17:15,195 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 60 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 64 - cumulative published bytes: 51983 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:17:15,196 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 3980 total (4 active) Queueing time: mean = 577.326 us, max = 225.583 ms, min = 727.000 ns, total = 2.298 s Execution time: mean = 312.190 us, total = 1.243 s Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 23.585 us, total = 28.350 ms, Queueing time: mean = 14.774 us, max = 6.895 ms, min = 1.848 us, total = 17.759 ms GcsInMemoryStore.Put - 527 total (0 active), Execution time: mean = 434.865 us, total = 229.174 ms, Queueing time: mean = 3.030 ms, max = 225.278 ms, min = 1.495 us, total = 1.597 s RayletLoadPulled - 300 total (1 active), Execution time: mean = 237.202 us, total = 71.161 ms, Queueing time: mean = 72.227 us, max = 1.351 ms, min = 10.621 us, total = 21.668 ms NodeManagerService.grpc_client.GetResourceLoad - 298 total (0 active), Execution time: mean = 1.322 ms, total = 393.886 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 298 total (0 active), Execution time: mean = 69.935 us, total = 20.841 ms, Queueing time: mean = 83.669 us, max = 230.850 us, min = 14.808 us, total = 24.933 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 135 total (0 active), Execution time: mean = 34.924 us, total = 4.715 ms, Queueing time: mean = 78.874 us, max = 239.072 us, min = 12.508 us, total = 10.648 ms InternalKVGcsService.grpc_server.InternalKVPut - 135 total (0 active), Execution time: mean = 482.874 us, total = 65.188 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 26.103 us, total = 3.132 ms, Queueing time: mean = 93.159 us, max = 277.841 us, min = 17.215 us, total = 11.179 ms GcsInMemoryStore.Get - 120 total (0 active), Execution time: mean = 18.959 us, total = 2.275 ms, Queueing time: mean = 16.237 us, max = 137.899 us, min = 2.227 us, total = 1.948 ms NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 506.209 us, total = 60.745 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet - 119 total (0 active), Execution time: mean = 1.121 ms, total = 133.447 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 119 total (0 active), Execution time: mean = 42.585 us, total = 5.068 ms, Queueing time: mean = 671.833 us, max = 22.982 ms, min = 9.997 us, total = 79.948 ms ClusterResourceManager.ResetRemoteNodeView - 101 total (1 active), Execution time: mean = 9.309 us, total = 940.226 us, Queueing time: mean = 68.987 us, max = 222.797 us, min = 27.397 us, total = 6.968 ms HealthCheck - 98 total (0 active), Execution time: mean = 7.220 us, total = 707.559 us, Queueing time: mean = 75.218 us, max = 173.947 us, min = 16.262 us, total = 7.371 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 79.951 us, total = 4.797 ms, Queueing time: mean = 94.327 us, max = 225.322 us, min = 22.663 us, total = 5.660 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 587.536 us, total = 35.252 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.272 ms, total = 38.153 ms, Queueing time: mean = 61.273 us, max = 123.501 us, min = 15.631 us, total = 1.838 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.285 ms, total = 6.424 ms, Queueing time: mean = 39.279 us, max = 67.424 us, min = 18.282 us, total = 196.393 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:17:15,196 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 13818 total (1 active) Queueing time: mean = 41.506 us, max = 395.058 us, min = 2.358 us, total = 573.535 ms Execution time: mean = 226.346 us, total = 3.128 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 6277 total (0 active), Execution time: mean = 467.311 us, total = 2.933 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6277 total (0 active), Execution time: mean = 26.158 us, total = 164.192 ms, Queueing time: mean = 88.987 us, max = 311.554 us, min = 5.968 us, total = 558.572 ms event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 24.571 us, total = 29.534 ms, Queueing time: mean = 9.040 us, max = 395.058 us, min = 2.358 us, total = 10.866 ms GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 9.840 us, total = 600.248 us, Queueing time: mean = 66.902 us, max = 214.030 us, min = 16.403 us, total = 4.081 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:17:15,196 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1559 total (25 active) Queueing time: mean = 17.786 us, max = 600.706 us, min = 2.413 us, total = 27.728 ms Execution time: mean = 189.520 ms, total = 295.462 s Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 26.346 us, total = 31.668 ms, Queueing time: mean = 10.033 us, max = 600.706 us, min = 2.413 us, total = 12.059 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 88 total (0 active), Execution time: mean = 37.047 us, total = 3.260 ms, Queueing time: mean = 86.652 us, max = 188.914 us, min = 8.714 us, total = 7.625 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 88 total (24 active), Execution time: mean = 3.356 s, total = 295.347 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 64 total (0 active), Execution time: mean = 103.992 us, total = 6.655 ms, Queueing time: mean = 93.709 us, max = 190.637 us, min = 21.208 us, total = 5.997 ms InternalPubSubGcsService.grpc_server.GcsPublish - 64 total (0 active), Execution time: mean = 588.872 us, total = 37.688 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 9.857 us, total = 19.713 us, Queueing time: mean = 27.194 us, max = 54.388 us, min = 54.388 us, total = 54.388 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:17:15,196 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1212 total (0 active) Queueing time: mean = 12.790 us, max = 1.999 ms, min = 693.000 ns, total = 15.501 ms Execution time: mean = 28.598 us, total = 34.660 ms Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 28.444 us, total = 34.190 ms, Queueing time: mean = 12.526 us, max = 1.999 ms, min = 4.195 us, total = 15.056 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:18:15,196 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 72 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 77 - cumulative published bytes: 61832 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:18:15,197 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 4699 total (4 active) Queueing time: mean = 495.478 us, max = 225.583 ms, min = 727.000 ns, total = 2.328 s Execution time: mean = 301.742 us, total = 1.418 s Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 24.009 us, total = 34.620 ms, Queueing time: mean = 13.971 us, max = 6.895 ms, min = 1.848 us, total = 20.146 ms GcsInMemoryStore.Put - 623 total (0 active), Execution time: mean = 368.872 us, total = 229.807 ms, Queueing time: mean = 2.571 ms, max = 225.278 ms, min = 1.495 us, total = 1.602 s RayletLoadPulled - 360 total (1 active), Execution time: mean = 241.375 us, total = 86.895 ms, Queueing time: mean = 73.089 us, max = 1.351 ms, min = 10.621 us, total = 26.312 ms NodeManagerService.grpc_client.GetResourceLoad - 358 total (0 active), Execution time: mean = 1.357 ms, total = 485.833 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 358 total (0 active), Execution time: mean = 71.347 us, total = 25.542 ms, Queueing time: mean = 85.808 us, max = 230.850 us, min = 14.808 us, total = 30.719 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 159 total (0 active), Execution time: mean = 35.868 us, total = 5.703 ms, Queueing time: mean = 84.290 us, max = 239.072 us, min = 12.508 us, total = 13.402 ms InternalKVGcsService.grpc_server.InternalKVPut - 159 total (0 active), Execution time: mean = 495.801 us, total = 78.832 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 27.598 us, total = 3.974 ms, Queueing time: mean = 95.526 us, max = 277.841 us, min = 17.215 us, total = 13.756 ms NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 513.086 us, total = 73.884 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 132 total (0 active), Execution time: mean = 19.322 us, total = 2.550 ms, Queueing time: mean = 15.897 us, max = 137.899 us, min = 2.227 us, total = 2.098 ms InternalKVGcsService.grpc_server.InternalKVGet - 131 total (0 active), Execution time: mean = 1.070 ms, total = 140.193 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 131 total (0 active), Execution time: mean = 43.335 us, total = 5.677 ms, Queueing time: mean = 619.987 us, max = 22.982 ms, min = 9.997 us, total = 81.218 ms ClusterResourceManager.ResetRemoteNodeView - 121 total (1 active), Execution time: mean = 9.527 us, total = 1.153 ms, Queueing time: mean = 72.612 us, max = 298.602 us, min = 27.397 us, total = 8.786 ms HealthCheck - 118 total (0 active), Execution time: mean = 7.543 us, total = 890.055 us, Queueing time: mean = 80.103 us, max = 178.553 us, min = 16.262 us, total = 9.452 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 81.344 us, total = 5.857 ms, Queueing time: mean = 98.072 us, max = 225.322 us, min = 22.663 us, total = 7.061 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 603.642 us, total = 43.462 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.297 ms, total = 46.678 ms, Queueing time: mean = 61.091 us, max = 123.501 us, min = 15.631 us, total = 2.199 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.345 ms, total = 8.070 ms, Queueing time: mean = 52.940 us, max = 121.246 us, min = 18.282 us, total = 317.639 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:18:15,197 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 16586 total (1 active) Queueing time: mean = 42.767 us, max = 2.737 ms, min = 2.358 us, total = 709.327 ms Execution time: mean = 231.094 us, total = 3.833 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 7535 total (0 active), Execution time: mean = 477.896 us, total = 3.601 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7535 total (0 active), Execution time: mean = 25.938 us, total = 195.445 ms, Queueing time: mean = 91.379 us, max = 311.554 us, min = 5.968 us, total = 688.538 ms event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 24.817 us, total = 35.786 ms, Queueing time: mean = 9.103 us, max = 395.058 us, min = 2.358 us, total = 13.126 ms GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 10.059 us, total = 734.295 us, Queueing time: mean = 104.751 us, max = 2.737 ms, min = 16.403 us, total = 7.647 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:18:15,198 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1851 total (25 active) Queueing time: mean = 17.997 us, max = 600.706 us, min = 2.413 us, total = 33.313 ms Execution time: mean = 192.138 ms, total = 355.647 s Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 26.965 us, total = 38.884 ms, Queueing time: mean = 10.176 us, max = 600.706 us, min = 2.413 us, total = 14.673 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 101 total (0 active), Execution time: mean = 38.230 us, total = 3.861 ms, Queueing time: mean = 89.238 us, max = 209.793 us, min = 8.714 us, total = 9.013 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 101 total (24 active), Execution time: mean = 3.520 s, total = 355.514 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 77 total (0 active), Execution time: mean = 103.401 us, total = 7.962 ms, Queueing time: mean = 98.456 us, max = 190.637 us, min = 21.208 us, total = 7.581 ms InternalPubSubGcsService.grpc_server.GcsPublish - 77 total (0 active), Execution time: mean = 601.890 us, total = 46.346 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 9.857 us, total = 19.713 us, Queueing time: mean = 27.194 us, max = 54.388 us, min = 54.388 us, total = 54.388 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:18:15,198 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1452 total (0 active) Queueing time: mean = 12.492 us, max = 1.999 ms, min = 693.000 ns, total = 18.138 ms Execution time: mean = 28.691 us, total = 41.659 ms Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 28.563 us, total = 41.188 ms, Queueing time: mean = 12.269 us, max = 1.999 ms, min = 4.195 us, total = 17.692 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:19:15,198 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 84 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 89 - cumulative published bytes: 71648 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 5 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 3 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 2 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:19:15,199 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 5418 total (4 active) Queueing time: mean = 435.466 us, max = 225.583 ms, min = 727.000 ns, total = 2.359 s Execution time: mean = 294.158 us, total = 1.594 s Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 24.132 us, total = 40.589 ms, Queueing time: mean = 13.375 us, max = 6.895 ms, min = 1.848 us, total = 22.497 ms GcsInMemoryStore.Put - 719 total (0 active), Execution time: mean = 320.440 us, total = 230.396 ms, Queueing time: mean = 2.234 ms, max = 225.278 ms, min = 1.495 us, total = 1.606 s RayletLoadPulled - 420 total (1 active), Execution time: mean = 244.231 us, total = 102.577 ms, Queueing time: mean = 74.219 us, max = 1.351 ms, min = 10.621 us, total = 31.172 ms NodeManagerService.grpc_client.GetResourceLoad - 418 total (0 active), Execution time: mean = 1.382 ms, total = 577.684 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 418 total (0 active), Execution time: mean = 72.621 us, total = 30.356 ms, Queueing time: mean = 89.644 us, max = 250.250 us, min = 14.808 us, total = 37.471 ms InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 183 total (0 active), Execution time: mean = 36.249 us, total = 6.634 ms, Queueing time: mean = 88.678 us, max = 239.072 us, min = 12.508 us, total = 16.228 ms InternalKVGcsService.grpc_server.InternalKVPut - 183 total (0 active), Execution time: mean = 507.485 us, total = 92.870 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 168 total (0 active), Execution time: mean = 28.932 us, total = 4.861 ms, Queueing time: mean = 99.274 us, max = 277.841 us, min = 17.215 us, total = 16.678 ms NodeInfoGcsService.grpc_server.CheckAlive - 168 total (0 active), Execution time: mean = 519.107 us, total = 87.210 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 144 total (0 active), Execution time: mean = 19.710 us, total = 2.838 ms, Queueing time: mean = 15.569 us, max = 137.899 us, min = 2.227 us, total = 2.242 ms InternalKVGcsService.grpc_server.InternalKVGet - 143 total (0 active), Execution time: mean = 1.030 ms, total = 147.309 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 143 total (0 active), Execution time: mean = 44.545 us, total = 6.370 ms, Queueing time: mean = 575.922 us, max = 22.982 ms, min = 9.997 us, total = 82.357 ms ClusterResourceManager.ResetRemoteNodeView - 141 total (1 active), Execution time: mean = 9.759 us, total = 1.376 ms, Queueing time: mean = 74.880 us, max = 298.602 us, min = 27.397 us, total = 10.558 ms HealthCheck - 138 total (0 active), Execution time: mean = 7.687 us, total = 1.061 ms, Queueing time: mean = 86.172 us, max = 178.553 us, min = 16.262 us, total = 11.892 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 84 total (0 active), Execution time: mean = 82.940 us, total = 6.967 ms, Queueing time: mean = 100.277 us, max = 225.322 us, min = 22.663 us, total = 8.423 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 84 total (0 active), Execution time: mean = 614.743 us, total = 51.638 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), Execution time: mean = 1.318 ms, total = 55.337 ms, Queueing time: mean = 64.203 us, max = 123.501 us, min = 15.631 us, total = 2.697 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), Execution time: mean = 1.345 ms, total = 9.413 ms, Queueing time: mean = 53.982 us, max = 121.246 us, min = 18.282 us, total = 377.873 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms GcsResourceManager::Update - 4 total (0 active), Execution time: mean = 92.418 us, total = 369.670 us, Queueing time: mean = 42.649 us, max = 90.090 us, min = 20.234 us, total = 170.597 us InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:19:15,199 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 19352 total (1 active) Queueing time: mean = 43.922 us, max = 2.737 ms, min = 2.358 us, total = 849.977 ms Execution time: mean = 235.281 us, total = 4.553 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 8792 total (0 active), Execution time: mean = 486.846 us, total = 4.280 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 8792 total (0 active), Execution time: mean = 26.183 us, total = 230.198 ms, Queueing time: mean = 93.966 us, max = 311.554 us, min = 5.968 us, total = 826.149 ms event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 24.816 us, total = 41.740 ms, Queueing time: mean = 9.093 us, max = 395.058 us, min = 2.358 us, total = 15.294 ms GcsTaskManager.GcJobSummary - 85 total (1 active), Execution time: mean = 10.189 us, total = 866.045 us, Queueing time: mean = 100.210 us, max = 2.737 ms, min = 16.403 us, total = 8.518 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:19:15,199 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2139 total (25 active) Queueing time: mean = 18.075 us, max = 600.706 us, min = 2.413 us, total = 38.662 ms Execution time: mean = 194.397 ms, total = 415.816 s Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 27.375 us, total = 46.045 ms, Queueing time: mean = 10.286 us, max = 600.706 us, min = 2.413 us, total = 17.302 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 113 total (0 active), Execution time: mean = 38.923 us, total = 4.398 ms, Queueing time: mean = 90.820 us, max = 209.793 us, min = 8.714 us, total = 10.263 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 113 total (24 active), Execution time: mean = 3.678 s, total = 415.665 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 89 total (0 active), Execution time: mean = 108.047 us, total = 9.616 ms, Queueing time: mean = 101.706 us, max = 190.637 us, min = 21.208 us, total = 9.052 ms InternalPubSubGcsService.grpc_server.GcsPublish - 89 total (0 active), Execution time: mean = 619.801 us, total = 55.162 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 9.857 us, total = 19.713 us, Queueing time: mean = 27.194 us, max = 54.388 us, min = 54.388 us, total = 54.388 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:19:15,199 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1692 total (0 active) Queueing time: mean = 12.244 us, max = 1.999 ms, min = 693.000 ns, total = 20.717 ms Execution time: mean = 28.640 us, total = 48.458 ms Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 28.530 us, total = 47.988 ms, Queueing time: mean = 12.052 us, max = 1.999 ms, min = 4.195 us, total = 20.272 ms - 4 total (0 active), Execution time: mean = 94.294 us, total = 377.177 us, Queueing time: mean = 44.991 us, max = 112.965 us, min = 15.988 us, total = 179.963 us RaySyncer.BroadcastMessage - 4 total (0 active), Execution time: mean = 23.074 us, total = 92.295 us, Queueing time: mean = 861.750 ns, max = 1.108 us, min = 693.000 ns, total = 3.447 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:20:15,199 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 96 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 104 - cumulative published bytes: 83067 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 12 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 5 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 4 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:20:15,201 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 6140 total (4 active) Queueing time: mean = 389.354 us, max = 225.583 ms, min = 727.000 ns, total = 2.391 s Execution time: mean = 287.607 us, total = 1.766 s Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 24.372 us, total = 46.842 ms, Queueing time: mean = 12.962 us, max = 6.895 ms, min = 1.848 us, total = 24.912 ms GcsInMemoryStore.Put - 815 total (0 active), Execution time: mean = 283.310 us, total = 230.897 ms, Queueing time: mean = 1.976 ms, max = 225.278 ms, min = 1.495 us, total = 1.610 s RayletLoadPulled - 480 total (1 active), Execution time: mean = 246.370 us, total = 118.258 ms, Queueing time: mean = 75.352 us, max = 1.351 ms, min = 10.621 us, total = 36.169 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 478 total (0 active), Execution time: mean = 73.017 us, total = 34.902 ms, Queueing time: mean = 92.320 us, max = 250.250 us, min = 14.808 us, total = 44.129 ms NodeManagerService.grpc_client.GetResourceLoad - 478 total (0 active), Execution time: mean = 1.393 ms, total = 665.842 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 207 total (0 active), Execution time: mean = 511.463 us, total = 105.873 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 207 total (0 active), Execution time: mean = 37.294 us, total = 7.720 ms, Queueing time: mean = 90.975 us, max = 239.072 us, min = 12.508 us, total = 18.832 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 192 total (0 active), Execution time: mean = 28.975 us, total = 5.563 ms, Queueing time: mean = 103.556 us, max = 277.841 us, min = 17.215 us, total = 19.883 ms NodeInfoGcsService.grpc_server.CheckAlive - 192 total (0 active), Execution time: mean = 523.493 us, total = 100.511 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 161 total (1 active), Execution time: mean = 9.968 us, total = 1.605 ms, Queueing time: mean = 73.269 us, max = 298.602 us, min = 27.397 us, total = 11.796 ms HealthCheck - 158 total (0 active), Execution time: mean = 7.679 us, total = 1.213 ms, Queueing time: mean = 88.850 us, max = 178.553 us, min = 16.262 us, total = 14.038 ms GcsInMemoryStore.Get - 156 total (0 active), Execution time: mean = 19.762 us, total = 3.083 ms, Queueing time: mean = 15.538 us, max = 137.899 us, min = 2.227 us, total = 2.424 ms InternalKVGcsService.grpc_server.InternalKVGet - 155 total (0 active), Execution time: mean = 996.386 us, total = 154.440 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 155 total (0 active), Execution time: mean = 46.408 us, total = 7.193 ms, Queueing time: mean = 540.364 us, max = 22.982 ms, min = 9.997 us, total = 83.756 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 96 total (0 active), Execution time: mean = 622.305 us, total = 59.741 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 96 total (0 active), Execution time: mean = 85.749 us, total = 8.232 ms, Queueing time: mean = 103.895 us, max = 225.322 us, min = 22.663 us, total = 9.974 ms GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), Execution time: mean = 1.341 ms, total = 64.370 ms, Queueing time: mean = 64.805 us, max = 123.501 us, min = 15.631 us, total = 3.111 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), Execution time: mean = 1.362 ms, total = 10.894 ms, Queueing time: mean = 57.689 us, max = 121.246 us, min = 18.282 us, total = 461.510 us GcsResourceManager::Update - 7 total (0 active), Execution time: mean = 118.669 us, total = 830.686 us, Queueing time: mean = 73.011 us, max = 155.007 us, min = 20.234 us, total = 511.078 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:20:15,201 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 22116 total (1 active) Queueing time: mean = 44.124 us, max = 2.737 ms, min = 2.358 us, total = 975.840 ms Execution time: mean = 235.759 us, total = 5.214 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 10048 total (0 active), Execution time: mean = 487.951 us, total = 4.903 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 10048 total (0 active), Execution time: mean = 26.109 us, total = 262.348 ms, Queueing time: mean = 94.452 us, max = 311.554 us, min = 5.968 us, total = 949.049 ms event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 24.852 us, total = 47.765 ms, Queueing time: mean = 9.029 us, max = 395.058 us, min = 2.358 us, total = 17.355 ms GcsTaskManager.GcJobSummary - 97 total (1 active), Execution time: mean = 10.104 us, total = 980.118 us, Queueing time: mean = 97.121 us, max = 2.737 ms, min = 16.403 us, total = 9.421 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:20:15,201 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2439 total (25 active) Queueing time: mean = 18.139 us, max = 600.706 us, min = 2.413 us, total = 44.241 ms Execution time: mean = 195.144 ms, total = 475.956 s Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 27.539 us, total = 52.930 ms, Queueing time: mean = 10.292 us, max = 600.706 us, min = 2.413 us, total = 19.781 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 128 total (0 active), Execution time: mean = 38.952 us, total = 4.986 ms, Queueing time: mean = 91.180 us, max = 209.793 us, min = 8.714 us, total = 11.671 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 128 total (24 active), Execution time: mean = 3.717 s, total = 475.787 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 104 total (0 active), Execution time: mean = 108.977 us, total = 11.334 ms, Queueing time: mean = 103.302 us, max = 190.637 us, min = 17.795 us, total = 10.743 ms InternalPubSubGcsService.grpc_server.GcsPublish - 104 total (0 active), Execution time: mean = 614.217 us, total = 63.879 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 9.857 us, total = 19.713 us, Queueing time: mean = 27.194 us, max = 54.388 us, min = 54.388 us, total = 54.388 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:20:15,201 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1938 total (0 active) Queueing time: mean = 12.250 us, max = 1.999 ms, min = 693.000 ns, total = 23.740 ms Execution time: mean = 28.878 us, total = 55.965 ms Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 28.583 us, total = 54.936 ms, Queueing time: mean = 11.912 us, max = 1.999 ms, min = 4.195 us, total = 22.895 ms - 7 total (0 active), Execution time: mean = 120.363 us, total = 842.540 us, Queueing time: mean = 82.284 us, max = 141.223 us, min = 15.988 us, total = 575.989 us RaySyncer.BroadcastMessage - 7 total (0 active), Execution time: mean = 26.485 us, total = 185.395 us, Queueing time: mean = 930.714 ns, max = 1.123 us, min = 693.000 ns, total = 6.515 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:21:15,201 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 108 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 119 - cumulative published bytes: 94606 - current buffered bytes: 818 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 20 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 7 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 6 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:21:15,202 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 6862 total (4 active) Queueing time: mean = 352.776 us, max = 225.583 ms, min = 727.000 ns, total = 2.421 s Execution time: mean = 283.963 us, total = 1.949 s Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 24.580 us, total = 53.142 ms, Queueing time: mean = 12.634 us, max = 6.895 ms, min = 1.848 us, total = 27.315 ms GcsInMemoryStore.Put - 911 total (0 active), Execution time: mean = 254.053 us, total = 231.443 ms, Queueing time: mean = 1.772 ms, max = 225.278 ms, min = 1.495 us, total = 1.614 s RayletLoadPulled - 540 total (1 active), Execution time: mean = 251.013 us, total = 135.547 ms, Queueing time: mean = 76.623 us, max = 1.351 ms, min = 10.621 us, total = 41.376 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 538 total (0 active), Execution time: mean = 73.974 us, total = 39.798 ms, Queueing time: mean = 93.755 us, max = 250.250 us, min = 14.808 us, total = 50.440 ms NodeManagerService.grpc_client.GetResourceLoad - 538 total (0 active), Execution time: mean = 1.414 ms, total = 760.622 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 231 total (0 active), Execution time: mean = 522.361 us, total = 120.665 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 231 total (0 active), Execution time: mean = 40.079 us, total = 9.258 ms, Queueing time: mean = 92.338 us, max = 239.072 us, min = 12.508 us, total = 21.330 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 216 total (0 active), Execution time: mean = 30.914 us, total = 6.677 ms, Queueing time: mean = 104.648 us, max = 277.841 us, min = 17.215 us, total = 22.604 ms NodeInfoGcsService.grpc_server.CheckAlive - 216 total (0 active), Execution time: mean = 527.894 us, total = 114.025 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 181 total (1 active), Execution time: mean = 9.864 us, total = 1.785 ms, Queueing time: mean = 73.547 us, max = 298.602 us, min = 22.681 us, total = 13.312 ms HealthCheck - 178 total (0 active), Execution time: mean = 7.765 us, total = 1.382 ms, Queueing time: mean = 88.993 us, max = 178.553 us, min = 16.262 us, total = 15.841 ms GcsInMemoryStore.Get - 168 total (0 active), Execution time: mean = 19.900 us, total = 3.343 ms, Queueing time: mean = 15.289 us, max = 137.899 us, min = 2.227 us, total = 2.569 ms InternalKVGcsService.grpc_server.InternalKVGet - 167 total (0 active), Execution time: mean = 968.614 us, total = 161.759 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 167 total (0 active), Execution time: mean = 48.662 us, total = 8.126 ms, Queueing time: mean = 508.605 us, max = 22.982 ms, min = 9.997 us, total = 84.937 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 108 total (0 active), Execution time: mean = 631.066 us, total = 68.155 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 108 total (0 active), Execution time: mean = 87.628 us, total = 9.464 ms, Queueing time: mean = 106.169 us, max = 225.322 us, min = 22.663 us, total = 11.466 ms GCSServer.deadline_timer.debug_state_dump - 54 total (1 active), Execution time: mean = 1.326 ms, total = 71.618 ms, Queueing time: mean = 65.303 us, max = 137.584 us, min = 15.631 us, total = 3.526 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 10 total (0 active), Execution time: mean = 111.823 us, total = 1.118 ms, Queueing time: mean = 79.910 us, max = 155.007 us, min = 20.234 us, total = 799.100 us GCSServer.deadline_timer.debug_state_event_stats_print - 9 total (1 active, 1 running), Execution time: mean = 1.415 ms, total = 12.734 ms, Queueing time: mean = 63.290 us, max = 121.246 us, min = 18.282 us, total = 569.612 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:21:15,202 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 24888 total (1 active) Queueing time: mean = 45.067 us, max = 2.737 ms, min = 2.358 us, total = 1.122 s Execution time: mean = 238.999 us, total = 5.948 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 11308 total (0 active), Execution time: mean = 494.636 us, total = 5.593 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 11308 total (0 active), Execution time: mean = 26.528 us, total = 299.975 ms, Queueing time: mean = 96.553 us, max = 311.554 us, min = 5.968 us, total = 1.092 s event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 24.878 us, total = 53.787 ms, Queueing time: mean = 9.024 us, max = 395.058 us, min = 2.358 us, total = 19.511 ms GcsTaskManager.GcJobSummary - 109 total (1 active), Execution time: mean = 10.009 us, total = 1.091 ms, Queueing time: mean = 94.331 us, max = 2.737 ms, min = 16.403 us, total = 10.282 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:21:15,203 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2739 total (25 active) Queueing time: mean = 18.407 us, max = 600.706 us, min = 2.413 us, total = 50.418 ms Execution time: mean = 195.760 ms, total = 536.186 s Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 27.594 us, total = 59.657 ms, Queueing time: mean = 10.277 us, max = 600.706 us, min = 2.413 us, total = 22.219 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 143 total (0 active), Execution time: mean = 39.646 us, total = 5.669 ms, Queueing time: mean = 94.249 us, max = 233.972 us, min = 8.714 us, total = 13.478 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 143 total (24 active), Execution time: mean = 3.748 s, total = 535.998 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 119 total (0 active), Execution time: mean = 107.148 us, total = 12.751 ms, Queueing time: mean = 106.514 us, max = 190.637 us, min = 17.795 us, total = 12.675 ms InternalPubSubGcsService.grpc_server.GcsPublish - 119 total (0 active), Execution time: mean = 618.415 us, total = 73.591 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 9.857 us, total = 19.713 us, Queueing time: mean = 27.194 us, max = 54.388 us, min = 54.388 us, total = 54.388 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:21:15,203 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2184 total (0 active) Queueing time: mean = 12.355 us, max = 1.999 ms, min = 693.000 ns, total = 26.983 ms Execution time: mean = 29.347 us, total = 64.093 ms Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 28.927 us, total = 62.539 ms, Queueing time: mean = 11.872 us, max = 1.999 ms, min = 4.195 us, total = 25.667 ms - 10 total (0 active), Execution time: mean = 128.357 us, total = 1.284 ms, Queueing time: mean = 104.512 us, max = 205.168 us, min = 15.988 us, total = 1.045 ms RaySyncer.BroadcastMessage - 10 total (0 active), Execution time: mean = 26.911 us, total = 269.107 us, Queueing time: mean = 895.500 ns, max = 1.123 us, min = 693.000 ns, total = 8.955 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:22:15,203 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 120 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 133 - cumulative published bytes: 106240 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 27 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 9 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 8 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:22:15,204 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 7583 total (4 active) Queueing time: mean = 323.371 us, max = 225.583 ms, min = 727.000 ns, total = 2.452 s Execution time: mean = 280.218 us, total = 2.125 s Event stats: event_loop_lag_probe - 2401 total (0 active), Execution time: mean = 24.684 us, total = 59.265 ms, Queueing time: mean = 12.336 us, max = 6.895 ms, min = 1.848 us, total = 29.619 ms GcsInMemoryStore.Put - 1007 total (0 active), Execution time: mean = 230.360 us, total = 231.973 ms, Queueing time: mean = 1.608 ms, max = 225.278 ms, min = 1.495 us, total = 1.619 s RayletLoadPulled - 600 total (1 active), Execution time: mean = 252.283 us, total = 151.370 ms, Queueing time: mean = 77.416 us, max = 1.351 ms, min = 10.621 us, total = 46.449 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 598 total (0 active), Execution time: mean = 75.280 us, total = 45.017 ms, Queueing time: mean = 95.445 us, max = 250.250 us, min = 14.808 us, total = 57.076 ms NodeManagerService.grpc_client.GetResourceLoad - 598 total (0 active), Execution time: mean = 1.426 ms, total = 852.837 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 255 total (0 active), Execution time: mean = 523.107 us, total = 133.392 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 255 total (0 active), Execution time: mean = 40.882 us, total = 10.425 ms, Queueing time: mean = 93.900 us, max = 239.072 us, min = 12.508 us, total = 23.945 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 240 total (0 active), Execution time: mean = 31.269 us, total = 7.505 ms, Queueing time: mean = 105.711 us, max = 277.841 us, min = 17.215 us, total = 25.371 ms NodeInfoGcsService.grpc_server.CheckAlive - 240 total (0 active), Execution time: mean = 532.925 us, total = 127.902 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 201 total (1 active), Execution time: mean = 9.985 us, total = 2.007 ms, Queueing time: mean = 73.821 us, max = 298.602 us, min = 19.990 us, total = 14.838 ms HealthCheck - 198 total (0 active), Execution time: mean = 7.815 us, total = 1.547 ms, Queueing time: mean = 90.399 us, max = 178.553 us, min = 16.262 us, total = 17.899 ms GcsInMemoryStore.Get - 180 total (0 active), Execution time: mean = 21.066 us, total = 3.792 ms, Queueing time: mean = 15.948 us, max = 137.899 us, min = 2.227 us, total = 2.871 ms InternalKVGcsService.grpc_server.InternalKVGet - 179 total (0 active), Execution time: mean = 944.551 us, total = 169.075 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 179 total (0 active), Execution time: mean = 50.000 us, total = 8.950 ms, Queueing time: mean = 481.458 us, max = 22.982 ms, min = 9.997 us, total = 86.181 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 120 total (0 active), Execution time: mean = 634.555 us, total = 76.147 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 120 total (0 active), Execution time: mean = 87.680 us, total = 10.522 ms, Queueing time: mean = 107.622 us, max = 225.322 us, min = 22.663 us, total = 12.915 ms GCSServer.deadline_timer.debug_state_dump - 60 total (1 active), Execution time: mean = 1.324 ms, total = 79.463 ms, Queueing time: mean = 65.475 us, max = 146.487 us, min = 15.631 us, total = 3.928 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 13 total (0 active), Execution time: mean = 110.174 us, total = 1.432 ms, Queueing time: mean = 83.372 us, max = 155.007 us, min = 20.234 us, total = 1.084 ms GCSServer.deadline_timer.debug_state_event_stats_print - 10 total (1 active, 1 running), Execution time: mean = 1.438 ms, total = 14.378 ms, Queueing time: mean = 60.571 us, max = 121.246 us, min = 18.282 us, total = 605.705 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:22:15,204 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 27657 total (1 active) Queueing time: mean = 45.568 us, max = 2.737 ms, min = 2.358 us, total = 1.260 s Execution time: mean = 240.685 us, total = 6.657 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 12567 total (0 active), Execution time: mean = 498.238 us, total = 6.261 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 12567 total (0 active), Execution time: mean = 26.597 us, total = 334.242 ms, Queueing time: mean = 97.680 us, max = 311.554 us, min = 5.968 us, total = 1.228 s event_loop_lag_probe - 2401 total (0 active), Execution time: mean = 24.903 us, total = 59.791 ms, Queueing time: mean = 8.995 us, max = 395.058 us, min = 2.358 us, total = 21.597 ms GcsTaskManager.GcJobSummary - 121 total (1 active), Execution time: mean = 10.022 us, total = 1.213 ms, Queueing time: mean = 91.904 us, max = 2.737 ms, min = 16.403 us, total = 11.120 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:22:15,204 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3081 total (25 active) Queueing time: mean = 19.029 us, max = 600.706 us, min = 2.413 us, total = 58.629 ms Execution time: mean = 4.657 s, total = 14347.947 s Event stats: event_loop_lag_probe - 2401 total (0 active), Execution time: mean = 27.751 us, total = 66.630 ms, Queueing time: mean = 10.310 us, max = 600.706 us, min = 2.413 us, total = 24.755 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 180 total (0 active), Execution time: mean = 37.508 us, total = 6.751 ms, Queueing time: mean = 96.406 us, max = 233.972 us, min = 8.714 us, total = 17.353 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 180 total (24 active), Execution time: mean = 79.710 s, total = 14347.739 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 133 total (0 active), Execution time: mean = 105.273 us, total = 14.001 ms, Queueing time: mean = 108.068 us, max = 190.637 us, min = 17.795 us, total = 14.373 ms InternalPubSubGcsService.grpc_server.GcsPublish - 133 total (0 active), Execution time: mean = 619.132 us, total = 82.345 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 631.149 us, total = 1.893 ms, Queueing time: mean = 52.199 us, max = 102.208 us, min = 54.388 us, total = 156.596 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:22:15,205 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2429 total (0 active) Queueing time: mean = 12.303 us, max = 1.999 ms, min = 563.000 ns, total = 29.884 ms Execution time: mean = 29.668 us, total = 72.063 ms Event stats: event_loop_lag_probe - 2401 total (0 active), Execution time: mean = 29.179 us, total = 70.059 ms, Queueing time: mean = 11.804 us, max = 1.999 ms, min = 4.195 us, total = 28.343 ms - 13 total (0 active), Execution time: mean = 127.994 us, total = 1.664 ms, Queueing time: mean = 97.586 us, max = 205.168 us, min = 15.988 us, total = 1.269 ms RaySyncer.BroadcastMessage - 13 total (0 active), Execution time: mean = 26.079 us, total = 339.026 us, Queueing time: mean = 858.692 ns, max = 1.123 us, min = 563.000 ns, total = 11.163 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:23:15,205 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 132 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 145 - cumulative published bytes: 116068 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 27 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 9 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 8 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:23:15,206 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 8302 total (4 active) Queueing time: mean = 298.675 us, max = 225.583 ms, min = 727.000 ns, total = 2.480 s Execution time: mean = 276.607 us, total = 2.296 s Event stats: event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 24.709 us, total = 65.258 ms, Queueing time: mean = 12.091 us, max = 6.895 ms, min = 1.268 us, total = 31.931 ms GcsInMemoryStore.Put - 1103 total (0 active), Execution time: mean = 210.740 us, total = 232.447 ms, Queueing time: mean = 1.471 ms, max = 225.278 ms, min = 1.495 us, total = 1.623 s RayletLoadPulled - 660 total (1 active), Execution time: mean = 250.803 us, total = 165.530 ms, Queueing time: mean = 77.406 us, max = 1.351 ms, min = 10.621 us, total = 51.088 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 658 total (0 active), Execution time: mean = 75.587 us, total = 49.736 ms, Queueing time: mean = 96.161 us, max = 250.250 us, min = 14.808 us, total = 63.274 ms NodeManagerService.grpc_client.GetResourceLoad - 658 total (0 active), Execution time: mean = 1.433 ms, total = 943.187 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 279 total (0 active), Execution time: mean = 525.583 us, total = 146.638 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 279 total (0 active), Execution time: mean = 42.318 us, total = 11.807 ms, Queueing time: mean = 93.431 us, max = 239.072 us, min = 12.508 us, total = 26.067 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 264 total (0 active), Execution time: mean = 31.847 us, total = 8.408 ms, Queueing time: mean = 104.422 us, max = 277.841 us, min = 17.215 us, total = 27.567 ms NodeInfoGcsService.grpc_server.CheckAlive - 264 total (0 active), Execution time: mean = 533.599 us, total = 140.870 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 221 total (1 active), Execution time: mean = 9.963 us, total = 2.202 ms, Queueing time: mean = 72.948 us, max = 298.602 us, min = 19.990 us, total = 16.122 ms HealthCheck - 218 total (0 active), Execution time: mean = 7.821 us, total = 1.705 ms, Queueing time: mean = 90.668 us, max = 178.553 us, min = 16.262 us, total = 19.766 ms GcsInMemoryStore.Get - 192 total (0 active), Execution time: mean = 22.038 us, total = 4.231 ms, Queueing time: mean = 15.701 us, max = 137.899 us, min = 2.227 us, total = 3.015 ms InternalKVGcsService.grpc_server.InternalKVGet - 191 total (0 active), Execution time: mean = 923.793 us, total = 176.445 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 191 total (0 active), Execution time: mean = 50.724 us, total = 9.688 ms, Queueing time: mean = 457.064 us, max = 22.982 ms, min = 9.997 us, total = 87.299 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 132 total (0 active), Execution time: mean = 635.281 us, total = 83.857 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 132 total (0 active), Execution time: mean = 88.158 us, total = 11.637 ms, Queueing time: mean = 106.942 us, max = 225.322 us, min = 22.663 us, total = 14.116 ms GCSServer.deadline_timer.debug_state_dump - 66 total (1 active), Execution time: mean = 1.323 ms, total = 87.335 ms, Queueing time: mean = 65.896 us, max = 146.487 us, min = 15.631 us, total = 4.349 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 13 total (0 active), Execution time: mean = 110.174 us, total = 1.432 ms, Queueing time: mean = 83.372 us, max = 155.007 us, min = 20.234 us, total = 1.084 ms GCSServer.deadline_timer.debug_state_event_stats_print - 11 total (1 active, 1 running), Execution time: mean = 1.462 ms, total = 16.086 ms, Queueing time: mean = 59.549 us, max = 121.246 us, min = 18.282 us, total = 655.042 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:23:15,206 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 30411 total (1 active) Queueing time: mean = 46.180 us, max = 2.737 ms, min = 2.358 us, total = 1.404 s Execution time: mean = 241.842 us, total = 7.355 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 13818 total (0 active), Execution time: mean = 500.774 us, total = 6.920 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 13818 total (0 active), Execution time: mean = 26.626 us, total = 367.924 ms, Queueing time: mean = 99.049 us, max = 1.100 ms, min = 5.968 us, total = 1.369 s event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 24.848 us, total = 65.625 ms, Queueing time: mean = 8.954 us, max = 395.058 us, min = 2.358 us, total = 23.647 ms GcsTaskManager.GcJobSummary - 133 total (1 active), Execution time: mean = 10.473 us, total = 1.393 ms, Queueing time: mean = 90.706 us, max = 2.737 ms, min = 16.403 us, total = 12.064 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:23:15,206 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3369 total (25 active) Queueing time: mean = 18.812 us, max = 600.706 us, min = 2.413 us, total = 63.377 ms Execution time: mean = 4.277 s, total = 14408.141 s Event stats: event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 27.710 us, total = 73.183 ms, Queueing time: mean = 10.233 us, max = 600.706 us, min = 2.413 us, total = 27.026 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 192 total (0 active), Execution time: mean = 38.076 us, total = 7.311 ms, Queueing time: mean = 97.000 us, max = 233.972 us, min = 8.714 us, total = 18.624 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 192 total (24 active), Execution time: mean = 75.041 s, total = 14407.917 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 145 total (0 active), Execution time: mean = 105.749 us, total = 15.334 ms, Queueing time: mean = 107.441 us, max = 190.637 us, min = 17.795 us, total = 15.579 ms InternalPubSubGcsService.grpc_server.GcsPublish - 145 total (0 active), Execution time: mean = 619.240 us, total = 89.790 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 631.149 us, total = 1.893 ms, Queueing time: mean = 52.199 us, max = 102.208 us, min = 54.388 us, total = 156.596 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:23:15,206 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2669 total (0 active) Queueing time: mean = 12.130 us, max = 1.999 ms, min = 563.000 ns, total = 32.376 ms Execution time: mean = 29.509 us, total = 78.758 ms Event stats: event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 29.063 us, total = 76.754 ms, Queueing time: mean = 11.675 us, max = 1.999 ms, min = 4.195 us, total = 30.834 ms - 13 total (0 active), Execution time: mean = 127.994 us, total = 1.664 ms, Queueing time: mean = 97.586 us, max = 205.168 us, min = 15.988 us, total = 1.269 ms RaySyncer.BroadcastMessage - 13 total (0 active), Execution time: mean = 26.079 us, total = 339.026 us, Queueing time: mean = 858.692 ns, max = 1.123 us, min = 563.000 ns, total = 11.163 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:24:15,206 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 144 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 157 - cumulative published bytes: 125896 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 27 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 9 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 8 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:24:15,207 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 9021 total (4 active) Queueing time: mean = 277.546 us, max = 225.583 ms, min = 727.000 ns, total = 2.504 s Execution time: mean = 271.534 us, total = 2.450 s Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 24.518 us, total = 70.637 ms, Queueing time: mean = 11.790 us, max = 6.895 ms, min = 1.268 us, total = 33.968 ms GcsInMemoryStore.Put - 1199 total (0 active), Execution time: mean = 194.223 us, total = 232.873 ms, Queueing time: mean = 1.356 ms, max = 225.278 ms, min = 1.495 us, total = 1.626 s RayletLoadPulled - 720 total (1 active), Execution time: mean = 247.999 us, total = 178.559 ms, Queueing time: mean = 77.032 us, max = 1.351 ms, min = 10.621 us, total = 55.463 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 718 total (0 active), Execution time: mean = 74.907 us, total = 53.783 ms, Queueing time: mean = 94.601 us, max = 250.250 us, min = 14.808 us, total = 67.923 ms NodeManagerService.grpc_client.GetResourceLoad - 718 total (0 active), Execution time: mean = 1.427 ms, total = 1.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 303 total (0 active), Execution time: mean = 521.317 us, total = 157.959 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 303 total (0 active), Execution time: mean = 41.593 us, total = 12.603 ms, Queueing time: mean = 91.504 us, max = 239.072 us, min = 12.508 us, total = 27.726 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 288 total (0 active), Execution time: mean = 31.736 us, total = 9.140 ms, Queueing time: mean = 104.294 us, max = 277.841 us, min = 17.215 us, total = 30.037 ms NodeInfoGcsService.grpc_server.CheckAlive - 288 total (0 active), Execution time: mean = 531.651 us, total = 153.115 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 241 total (1 active), Execution time: mean = 9.925 us, total = 2.392 ms, Queueing time: mean = 72.437 us, max = 298.602 us, min = 19.990 us, total = 17.457 ms HealthCheck - 238 total (0 active), Execution time: mean = 7.771 us, total = 1.850 ms, Queueing time: mean = 88.703 us, max = 178.553 us, min = 12.322 us, total = 21.111 ms GcsInMemoryStore.Get - 204 total (0 active), Execution time: mean = 22.338 us, total = 4.557 ms, Queueing time: mean = 15.443 us, max = 137.899 us, min = 2.227 us, total = 3.150 ms InternalKVGcsService.grpc_server.InternalKVGet - 203 total (0 active), Execution time: mean = 900.938 us, total = 182.890 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 203 total (0 active), Execution time: mean = 50.591 us, total = 10.270 ms, Queueing time: mean = 434.962 us, max = 22.982 ms, min = 9.997 us, total = 88.297 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 144 total (0 active), Execution time: mean = 627.089 us, total = 90.301 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 144 total (0 active), Execution time: mean = 86.761 us, total = 12.494 ms, Queueing time: mean = 106.377 us, max = 225.322 us, min = 22.663 us, total = 15.318 ms GCSServer.deadline_timer.debug_state_dump - 72 total (1 active), Execution time: mean = 1.311 ms, total = 94.369 ms, Queueing time: mean = 64.577 us, max = 146.487 us, min = 14.502 us, total = 4.650 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 13 total (0 active), Execution time: mean = 110.174 us, total = 1.432 ms, Queueing time: mean = 83.372 us, max = 155.007 us, min = 20.234 us, total = 1.084 ms GCSServer.deadline_timer.debug_state_event_stats_print - 12 total (1 active, 1 running), Execution time: mean = 1.474 ms, total = 17.684 ms, Queueing time: mean = 59.018 us, max = 121.246 us, min = 18.282 us, total = 708.215 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:24:15,208 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 33183 total (1 active) Queueing time: mean = 45.874 us, max = 2.737 ms, min = 2.124 us, total = 1.522 s Execution time: mean = 240.832 us, total = 7.992 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 15078 total (0 active), Execution time: mean = 499.055 us, total = 7.525 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 15078 total (0 active), Execution time: mean = 26.132 us, total = 394.012 ms, Queueing time: mean = 98.394 us, max = 1.100 ms, min = 4.653 us, total = 1.484 s event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 24.730 us, total = 71.248 ms, Queueing time: mean = 8.903 us, max = 395.058 us, min = 2.124 us, total = 25.649 ms GcsTaskManager.GcJobSummary - 145 total (1 active), Execution time: mean = 10.481 us, total = 1.520 ms, Queueing time: mean = 89.439 us, max = 2.737 ms, min = 16.403 us, total = 12.969 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:24:15,208 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3657 total (25 active) Queueing time: mean = 18.310 us, max = 600.706 us, min = 2.413 us, total = 66.959 ms Execution time: mean = 3.956 s, total = 14468.269 s Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 27.273 us, total = 78.572 ms, Queueing time: mean = 10.070 us, max = 600.706 us, min = 2.413 us, total = 29.012 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 204 total (0 active), Execution time: mean = 37.792 us, total = 7.710 ms, Queueing time: mean = 94.319 us, max = 233.972 us, min = 8.714 us, total = 19.241 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 204 total (24 active), Execution time: mean = 70.922 s, total = 14468.032 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 157 total (0 active), Execution time: mean = 104.961 us, total = 16.479 ms, Queueing time: mean = 105.465 us, max = 190.637 us, min = 17.795 us, total = 16.558 ms InternalPubSubGcsService.grpc_server.GcsPublish - 157 total (0 active), Execution time: mean = 613.565 us, total = 96.330 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 631.149 us, total = 1.893 ms, Queueing time: mean = 52.199 us, max = 102.208 us, min = 54.388 us, total = 156.596 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:24:15,208 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2909 total (0 active) Queueing time: mean = 11.870 us, max = 1.999 ms, min = 563.000 ns, total = 34.529 ms Execution time: mean = 29.095 us, total = 84.637 ms Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 28.682 us, total = 82.633 ms, Queueing time: mean = 11.450 us, max = 1.999 ms, min = 3.185 us, total = 32.988 ms - 13 total (0 active), Execution time: mean = 127.994 us, total = 1.664 ms, Queueing time: mean = 97.586 us, max = 205.168 us, min = 15.988 us, total = 1.269 ms RaySyncer.BroadcastMessage - 13 total (0 active), Execution time: mean = 26.079 us, total = 339.026 us, Queueing time: mean = 858.692 ns, max = 1.123 us, min = 563.000 ns, total = 11.163 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:25:15,208 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 156 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 169 - cumulative published bytes: 135724 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 27 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 9 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 8 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:25:15,209 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 9740 total (4 active) Queueing time: mean = 260.123 us, max = 225.583 ms, min = 727.000 ns, total = 2.534 s Execution time: mean = 269.542 us, total = 2.625 s Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 24.725 us, total = 77.165 ms, Queueing time: mean = 11.655 us, max = 6.895 ms, min = 1.268 us, total = 36.375 ms GcsInMemoryStore.Put - 1295 total (0 active), Execution time: mean = 180.213 us, total = 233.375 ms, Queueing time: mean = 1.259 ms, max = 225.278 ms, min = 1.495 us, total = 1.631 s RayletLoadPulled - 780 total (1 active), Execution time: mean = 249.101 us, total = 194.299 ms, Queueing time: mean = 77.339 us, max = 1.351 ms, min = 10.621 us, total = 60.324 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 778 total (0 active), Execution time: mean = 75.455 us, total = 58.704 ms, Queueing time: mean = 95.573 us, max = 250.250 us, min = 14.808 us, total = 74.356 ms NodeManagerService.grpc_client.GetResourceLoad - 778 total (0 active), Execution time: mean = 1.433 ms, total = 1.115 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 327 total (0 active), Execution time: mean = 525.048 us, total = 171.691 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 327 total (0 active), Execution time: mean = 41.834 us, total = 13.680 ms, Queueing time: mean = 92.356 us, max = 239.072 us, min = 12.508 us, total = 30.200 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 312 total (0 active), Execution time: mean = 32.206 us, total = 10.048 ms, Queueing time: mean = 104.634 us, max = 277.841 us, min = 17.215 us, total = 32.646 ms NodeInfoGcsService.grpc_server.CheckAlive - 312 total (0 active), Execution time: mean = 539.056 us, total = 168.186 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 261 total (1 active), Execution time: mean = 10.253 us, total = 2.676 ms, Queueing time: mean = 72.970 us, max = 298.602 us, min = 19.990 us, total = 19.045 ms HealthCheck - 258 total (0 active), Execution time: mean = 7.809 us, total = 2.015 ms, Queueing time: mean = 89.754 us, max = 238.751 us, min = 12.322 us, total = 23.157 ms GcsInMemoryStore.Get - 216 total (0 active), Execution time: mean = 22.201 us, total = 4.796 ms, Queueing time: mean = 15.400 us, max = 137.899 us, min = 2.227 us, total = 3.326 ms InternalKVGcsService.grpc_server.InternalKVGet - 215 total (0 active), Execution time: mean = 884.539 us, total = 190.176 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 215 total (0 active), Execution time: mean = 51.355 us, total = 11.041 ms, Queueing time: mean = 416.516 us, max = 22.982 ms, min = 9.997 us, total = 89.551 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 156 total (0 active), Execution time: mean = 632.174 us, total = 98.619 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 156 total (0 active), Execution time: mean = 87.990 us, total = 13.726 ms, Queueing time: mean = 106.850 us, max = 225.322 us, min = 22.663 us, total = 16.669 ms GCSServer.deadline_timer.debug_state_dump - 78 total (1 active), Execution time: mean = 1.298 ms, total = 101.239 ms, Queueing time: mean = 63.903 us, max = 146.487 us, min = 14.502 us, total = 4.984 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 13 total (0 active), Execution time: mean = 110.174 us, total = 1.432 ms, Queueing time: mean = 83.372 us, max = 155.007 us, min = 20.234 us, total = 1.084 ms GCSServer.deadline_timer.debug_state_event_stats_print - 13 total (1 active, 1 running), Execution time: mean = 1.487 ms, total = 19.328 ms, Queueing time: mean = 59.769 us, max = 121.246 us, min = 18.282 us, total = 776.997 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:25:15,210 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 35955 total (1 active) Queueing time: mean = 46.175 us, max = 2.737 ms, min = 2.124 us, total = 1.660 s Execution time: mean = 241.859 us, total = 8.696 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 16338 total (0 active), Execution time: mean = 501.349 us, total = 8.191 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 16338 total (0 active), Execution time: mean = 26.088 us, total = 426.226 ms, Queueing time: mean = 99.060 us, max = 1.100 ms, min = 4.653 us, total = 1.618 s event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 24.706 us, total = 77.106 ms, Queueing time: mean = 8.899 us, max = 395.058 us, min = 2.124 us, total = 27.774 ms GcsTaskManager.GcJobSummary - 157 total (1 active), Execution time: mean = 10.517 us, total = 1.651 ms, Queueing time: mean = 89.052 us, max = 2.737 ms, min = 16.403 us, total = 13.981 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:25:15,210 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3945 total (25 active) Queueing time: mean = 18.149 us, max = 600.706 us, min = 2.413 us, total = 71.599 ms Execution time: mean = 3.683 s, total = 14528.418 s Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 27.059 us, total = 84.453 ms, Queueing time: mean = 10.023 us, max = 600.706 us, min = 2.413 us, total = 31.282 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 216 total (0 active), Execution time: mean = 38.396 us, total = 8.293 ms, Queueing time: mean = 93.973 us, max = 233.972 us, min = 8.714 us, total = 20.298 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 216 total (24 active), Execution time: mean = 67.260 s, total = 14528.165 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 169 total (0 active), Execution time: mean = 106.638 us, total = 18.022 ms, Queueing time: mean = 105.747 us, max = 190.637 us, min = 17.795 us, total = 17.871 ms InternalPubSubGcsService.grpc_server.GcsPublish - 169 total (0 active), Execution time: mean = 615.636 us, total = 104.042 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 631.149 us, total = 1.893 ms, Queueing time: mean = 52.199 us, max = 102.208 us, min = 54.388 us, total = 156.596 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:25:15,210 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3149 total (0 active) Queueing time: mean = 11.785 us, max = 1.999 ms, min = 563.000 ns, total = 37.112 ms Execution time: mean = 29.077 us, total = 91.562 ms Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 28.695 us, total = 89.558 ms, Queueing time: mean = 11.397 us, max = 1.999 ms, min = 3.185 us, total = 35.570 ms - 13 total (0 active), Execution time: mean = 127.994 us, total = 1.664 ms, Queueing time: mean = 97.586 us, max = 205.168 us, min = 15.988 us, total = 1.269 ms RaySyncer.BroadcastMessage - 13 total (0 active), Execution time: mean = 26.079 us, total = 339.026 us, Queueing time: mean = 858.692 ns, max = 1.123 us, min = 563.000 ns, total = 11.163 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:26:15,210 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 168 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 183 - cumulative published bytes: 146936 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 35 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:26:15,211 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 10462 total (4 active) Queueing time: mean = 244.927 us, max = 225.583 ms, min = 727.000 ns, total = 2.562 s Execution time: mean = 267.795 us, total = 2.802 s Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 24.830 us, total = 83.454 ms, Queueing time: mean = 11.512 us, max = 6.895 ms, min = 1.268 us, total = 38.690 ms GcsInMemoryStore.Put - 1391 total (0 active), Execution time: mean = 168.096 us, total = 233.821 ms, Queueing time: mean = 1.175 ms, max = 225.278 ms, min = 1.495 us, total = 1.635 s RayletLoadPulled - 840 total (1 active), Execution time: mean = 249.749 us, total = 209.789 ms, Queueing time: mean = 77.438 us, max = 1.351 ms, min = 10.621 us, total = 65.048 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 838 total (0 active), Execution time: mean = 75.957 us, total = 63.652 ms, Queueing time: mean = 95.717 us, max = 250.250 us, min = 14.808 us, total = 80.211 ms NodeManagerService.grpc_client.GetResourceLoad - 838 total (0 active), Execution time: mean = 1.441 ms, total = 1.207 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 351 total (0 active), Execution time: mean = 527.048 us, total = 184.994 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 351 total (0 active), Execution time: mean = 41.719 us, total = 14.643 ms, Queueing time: mean = 92.019 us, max = 239.072 us, min = 12.508 us, total = 32.299 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 336 total (0 active), Execution time: mean = 33.013 us, total = 11.092 ms, Queueing time: mean = 105.982 us, max = 277.841 us, min = 17.215 us, total = 35.610 ms NodeInfoGcsService.grpc_server.CheckAlive - 336 total (0 active), Execution time: mean = 541.403 us, total = 181.911 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 281 total (1 active), Execution time: mean = 10.266 us, total = 2.885 ms, Queueing time: mean = 73.171 us, max = 298.602 us, min = 19.990 us, total = 20.561 ms HealthCheck - 278 total (0 active), Execution time: mean = 7.796 us, total = 2.167 ms, Queueing time: mean = 89.338 us, max = 238.751 us, min = 12.322 us, total = 24.836 ms GcsInMemoryStore.Get - 228 total (0 active), Execution time: mean = 22.031 us, total = 5.023 ms, Queueing time: mean = 15.268 us, max = 137.899 us, min = 2.227 us, total = 3.481 ms InternalKVGcsService.grpc_server.InternalKVGet - 227 total (0 active), Execution time: mean = 869.712 us, total = 197.425 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 227 total (0 active), Execution time: mean = 53.064 us, total = 12.046 ms, Queueing time: mean = 399.741 us, max = 22.982 ms, min = 9.997 us, total = 90.741 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 168 total (0 active), Execution time: mean = 636.686 us, total = 106.963 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 168 total (0 active), Execution time: mean = 91.128 us, total = 15.310 ms, Queueing time: mean = 106.344 us, max = 225.322 us, min = 22.663 us, total = 17.866 ms GCSServer.deadline_timer.debug_state_dump - 84 total (1 active), Execution time: mean = 1.289 ms, total = 108.299 ms, Queueing time: mean = 64.352 us, max = 146.487 us, min = 14.502 us, total = 5.406 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 16 total (0 active), Execution time: mean = 111.612 us, total = 1.786 ms, Queueing time: mean = 92.105 us, max = 159.376 us, min = 20.234 us, total = 1.474 ms GCSServer.deadline_timer.debug_state_event_stats_print - 14 total (1 active, 1 running), Execution time: mean = 1.512 ms, total = 21.164 ms, Queueing time: mean = 59.995 us, max = 121.246 us, min = 18.282 us, total = 839.931 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:26:15,211 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 38727 total (1 active) Queueing time: mean = 46.453 us, max = 2.737 ms, min = 2.124 us, total = 1.799 s Execution time: mean = 242.597 us, total = 9.395 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 17598 total (0 active), Execution time: mean = 502.890 us, total = 8.850 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 17598 total (0 active), Execution time: mean = 26.161 us, total = 460.387 ms, Queueing time: mean = 99.688 us, max = 1.100 ms, min = 4.263 us, total = 1.754 s event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 24.696 us, total = 83.004 ms, Queueing time: mean = 8.874 us, max = 395.058 us, min = 2.124 us, total = 29.825 ms GcsTaskManager.GcJobSummary - 169 total (1 active), Execution time: mean = 10.547 us, total = 1.782 ms, Queueing time: mean = 87.717 us, max = 2.737 ms, min = 16.403 us, total = 14.824 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:26:15,211 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4241 total (25 active) Queueing time: mean = 18.178 us, max = 600.706 us, min = 2.413 us, total = 77.092 ms Execution time: mean = 3.440 s, total = 14588.618 s Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 27.152 us, total = 91.258 ms, Queueing time: mean = 10.022 us, max = 600.706 us, min = 2.413 us, total = 33.683 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 230 total (0 active), Execution time: mean = 39.054 us, total = 8.983 ms, Queueing time: mean = 94.844 us, max = 233.972 us, min = 8.714 us, total = 21.814 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 230 total (24 active), Execution time: mean = 63.428 s, total = 14588.347 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 183 total (0 active), Execution time: mean = 107.121 us, total = 19.603 ms, Queueing time: mean = 106.270 us, max = 190.637 us, min = 17.795 us, total = 19.447 ms InternalPubSubGcsService.grpc_server.GcsPublish - 183 total (0 active), Execution time: mean = 618.209 us, total = 113.132 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 631.149 us, total = 1.893 ms, Queueing time: mean = 52.199 us, max = 102.208 us, min = 54.388 us, total = 156.596 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:26:15,212 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3395 total (0 active) Queueing time: mean = 11.762 us, max = 1.999 ms, min = 563.000 ns, total = 39.932 ms Execution time: mean = 29.112 us, total = 98.836 ms Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 28.631 us, total = 96.227 ms, Queueing time: mean = 11.317 us, max = 1.999 ms, min = 3.185 us, total = 38.037 ms - 16 total (0 active), Execution time: mean = 133.267 us, total = 2.132 ms, Queueing time: mean = 101.181 us, max = 205.168 us, min = 15.988 us, total = 1.619 ms RaySyncer.BroadcastMessage - 16 total (0 active), Execution time: mean = 29.699 us, total = 475.190 us, Queueing time: mean = 887.312 ns, max = 1.296 us, min = 563.000 ns, total = 14.197 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:27:15,212 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 180 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 197 - cumulative published bytes: 157066 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 42 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 13 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 12 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:27:15,212 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 11183 total (4 active) Queueing time: mean = 231.803 us, max = 225.583 ms, min = 727.000 ns, total = 2.592 s Execution time: mean = 266.346 us, total = 2.979 s Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 25.030 us, total = 90.132 ms, Queueing time: mean = 11.426 us, max = 6.895 ms, min = 1.268 us, total = 41.144 ms GcsInMemoryStore.Put - 1487 total (0 active), Execution time: mean = 157.562 us, total = 234.294 ms, Queueing time: mean = 1.102 ms, max = 225.278 ms, min = 1.495 us, total = 1.639 s RayletLoadPulled - 900 total (1 active), Execution time: mean = 249.957 us, total = 224.962 ms, Queueing time: mean = 77.260 us, max = 1.351 ms, min = 10.621 us, total = 69.534 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 898 total (0 active), Execution time: mean = 75.803 us, total = 68.071 ms, Queueing time: mean = 95.967 us, max = 250.250 us, min = 14.808 us, total = 86.178 ms NodeManagerService.grpc_client.GetResourceLoad - 898 total (0 active), Execution time: mean = 1.446 ms, total = 1.298 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 375 total (0 active), Execution time: mean = 531.554 us, total = 199.333 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 375 total (0 active), Execution time: mean = 41.531 us, total = 15.574 ms, Queueing time: mean = 93.745 us, max = 239.072 us, min = 12.508 us, total = 35.154 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 360 total (0 active), Execution time: mean = 33.042 us, total = 11.895 ms, Queueing time: mean = 107.223 us, max = 277.841 us, min = 17.215 us, total = 38.600 ms NodeInfoGcsService.grpc_server.CheckAlive - 360 total (0 active), Execution time: mean = 545.271 us, total = 196.297 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 301 total (1 active), Execution time: mean = 10.386 us, total = 3.126 ms, Queueing time: mean = 73.262 us, max = 298.602 us, min = 19.990 us, total = 22.052 ms HealthCheck - 298 total (0 active), Execution time: mean = 7.797 us, total = 2.323 ms, Queueing time: mean = 89.975 us, max = 238.751 us, min = 12.322 us, total = 26.813 ms GcsInMemoryStore.Get - 240 total (0 active), Execution time: mean = 22.180 us, total = 5.323 ms, Queueing time: mean = 15.160 us, max = 137.899 us, min = 2.227 us, total = 3.638 ms InternalKVGcsService.grpc_server.InternalKVGet - 239 total (0 active), Execution time: mean = 856.187 us, total = 204.629 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 239 total (0 active), Execution time: mean = 53.394 us, total = 12.761 ms, Queueing time: mean = 384.522 us, max = 22.982 ms, min = 9.997 us, total = 91.901 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 180 total (0 active), Execution time: mean = 640.453 us, total = 115.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 180 total (0 active), Execution time: mean = 90.994 us, total = 16.379 ms, Queueing time: mean = 107.062 us, max = 225.322 us, min = 22.663 us, total = 19.271 ms GCSServer.deadline_timer.debug_state_dump - 90 total (1 active), Execution time: mean = 1.301 ms, total = 117.101 ms, Queueing time: mean = 64.555 us, max = 146.487 us, min = 14.502 us, total = 5.810 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 18 total (0 active), Execution time: mean = 110.799 us, total = 1.994 ms, Queueing time: mean = 97.282 us, max = 175.233 us, min = 20.234 us, total = 1.751 ms GCSServer.deadline_timer.debug_state_event_stats_print - 15 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 22.810 ms, Queueing time: mean = 57.399 us, max = 121.246 us, min = 18.282 us, total = 860.978 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:27:15,212 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 41499 total (1 active) Queueing time: mean = 46.771 us, max = 3.642 ms, min = 2.014 us, total = 1.941 s Execution time: mean = 243.350 us, total = 10.099 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 18858 total (0 active), Execution time: mean = 504.508 us, total = 9.514 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 18858 total (0 active), Execution time: mean = 26.195 us, total = 493.976 ms, Queueing time: mean = 100.212 us, max = 1.100 ms, min = 4.263 us, total = 1.890 s event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 24.671 us, total = 88.842 ms, Queueing time: mean = 8.842 us, max = 395.058 us, min = 2.014 us, total = 31.841 ms GcsTaskManager.GcJobSummary - 181 total (1 active), Execution time: mean = 10.607 us, total = 1.920 ms, Queueing time: mean = 106.684 us, max = 3.642 ms, min = 16.403 us, total = 19.310 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:27:15,212 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4540 total (25 active) Queueing time: mean = 18.255 us, max = 600.706 us, min = 2.413 us, total = 82.880 ms Execution time: mean = 3.293 s, total = 14948.791 s Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 27.320 us, total = 98.379 ms, Queueing time: mean = 10.060 us, max = 600.706 us, min = 2.413 us, total = 36.227 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 245 total (0 active), Execution time: mean = 39.194 us, total = 9.603 ms, Queueing time: mean = 95.633 us, max = 233.972 us, min = 8.714 us, total = 23.430 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 245 total (24 active), Execution time: mean = 61.014 s, total = 14948.501 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 197 total (0 active), Execution time: mean = 108.179 us, total = 21.311 ms, Queueing time: mean = 106.398 us, max = 190.637 us, min = 17.795 us, total = 20.960 ms InternalPubSubGcsService.grpc_server.GcsPublish - 197 total (0 active), Execution time: mean = 620.962 us, total = 122.329 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 488.179 us, total = 1.953 ms, Queueing time: mean = 67.763 us, max = 114.456 us, min = 54.388 us, total = 271.052 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:27:15,213 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3639 total (0 active) Queueing time: mean = 11.762 us, max = 1.999 ms, min = 563.000 ns, total = 42.801 ms Execution time: mean = 29.310 us, total = 106.661 ms Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 28.792 us, total = 103.681 ms, Queueing time: mean = 11.288 us, max = 1.999 ms, min = 3.185 us, total = 40.647 ms - 18 total (0 active), Execution time: mean = 135.160 us, total = 2.433 ms, Queueing time: mean = 104.159 us, max = 205.168 us, min = 15.988 us, total = 1.875 ms RaySyncer.BroadcastMessage - 18 total (0 active), Execution time: mean = 30.343 us, total = 546.176 us, Queueing time: mean = 930.389 ns, max = 1.380 us, min = 563.000 ns, total = 16.747 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:28:15,213 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 192 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 211 - cumulative published bytes: 167199 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 44 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 13 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 12 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:28:15,214 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 11903 total (4 active) Queueing time: mean = 220.021 us, max = 225.583 ms, min = 727.000 ns, total = 2.619 s Execution time: mean = 262.733 us, total = 3.127 s Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 24.888 us, total = 95.596 ms, Queueing time: mean = 11.245 us, max = 6.895 ms, min = 1.268 us, total = 43.191 ms GcsInMemoryStore.Put - 1583 total (0 active), Execution time: mean = 148.268 us, total = 234.708 ms, Queueing time: mean = 1.037 ms, max = 225.278 ms, min = 1.495 us, total = 1.642 s RayletLoadPulled - 960 total (1 active), Execution time: mean = 247.840 us, total = 237.926 ms, Queueing time: mean = 76.219 us, max = 1.351 ms, min = 10.621 us, total = 73.171 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 958 total (0 active), Execution time: mean = 75.007 us, total = 71.856 ms, Queueing time: mean = 99.449 us, max = 5.512 ms, min = 11.600 us, total = 95.272 ms NodeManagerService.grpc_client.GetResourceLoad - 958 total (0 active), Execution time: mean = 1.438 ms, total = 1.377 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 399 total (0 active), Execution time: mean = 528.965 us, total = 211.057 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 399 total (0 active), Execution time: mean = 40.944 us, total = 16.337 ms, Queueing time: mean = 92.972 us, max = 239.072 us, min = 12.508 us, total = 37.096 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 384 total (0 active), Execution time: mean = 32.666 us, total = 12.544 ms, Queueing time: mean = 104.673 us, max = 277.841 us, min = 17.215 us, total = 40.194 ms NodeInfoGcsService.grpc_server.CheckAlive - 384 total (0 active), Execution time: mean = 538.238 us, total = 206.683 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 321 total (1 active), Execution time: mean = 10.283 us, total = 3.301 ms, Queueing time: mean = 73.214 us, max = 298.602 us, min = 19.990 us, total = 23.502 ms HealthCheck - 318 total (0 active), Execution time: mean = 7.723 us, total = 2.456 ms, Queueing time: mean = 87.987 us, max = 238.751 us, min = 12.322 us, total = 27.980 ms GcsInMemoryStore.Get - 252 total (0 active), Execution time: mean = 21.846 us, total = 5.505 ms, Queueing time: mean = 14.872 us, max = 137.899 us, min = 2.227 us, total = 3.748 ms InternalKVGcsService.grpc_server.InternalKVGet - 251 total (0 active), Execution time: mean = 840.273 us, total = 210.909 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 251 total (0 active), Execution time: mean = 53.206 us, total = 13.355 ms, Queueing time: mean = 369.532 us, max = 22.982 ms, min = 9.997 us, total = 92.753 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 192 total (0 active), Execution time: mean = 635.376 us, total = 121.992 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 192 total (0 active), Execution time: mean = 90.792 us, total = 17.432 ms, Queueing time: mean = 105.763 us, max = 225.322 us, min = 22.663 us, total = 20.306 ms GCSServer.deadline_timer.debug_state_dump - 96 total (1 active), Execution time: mean = 1.297 ms, total = 124.477 ms, Queueing time: mean = 64.707 us, max = 159.461 us, min = 14.502 us, total = 6.212 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 19 total (0 active), Execution time: mean = 109.540 us, total = 2.081 ms, Queueing time: mean = 94.057 us, max = 175.233 us, min = 20.234 us, total = 1.787 ms GCSServer.deadline_timer.debug_state_event_stats_print - 16 total (1 active, 1 running), Execution time: mean = 1.485 ms, total = 23.765 ms, Queueing time: mean = 56.243 us, max = 121.246 us, min = 18.282 us, total = 899.881 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:28:15,214 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 44271 total (1 active) Queueing time: mean = 46.492 us, max = 3.642 ms, min = 2.014 us, total = 2.058 s Execution time: mean = 242.012 us, total = 10.714 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 20118 total (0 active), Execution time: mean = 501.823 us, total = 10.096 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 20118 total (0 active), Execution time: mean = 25.958 us, total = 522.227 ms, Queueing time: mean = 99.632 us, max = 1.100 ms, min = 4.024 us, total = 2.004 s event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 24.517 us, total = 94.170 ms, Queueing time: mean = 8.781 us, max = 395.058 us, min = 2.014 us, total = 33.729 ms GcsTaskManager.GcJobSummary - 193 total (1 active), Execution time: mean = 10.506 us, total = 2.028 ms, Queueing time: mean = 104.233 us, max = 3.642 ms, min = 16.403 us, total = 20.117 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:28:15,214 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4836 total (25 active) Queueing time: mean = 18.142 us, max = 600.706 us, min = 2.413 us, total = 87.734 ms Execution time: mean = 3.104 s, total = 15009.005 s Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 27.297 us, total = 104.847 ms, Queueing time: mean = 10.033 us, max = 600.706 us, min = 2.413 us, total = 38.537 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 259 total (0 active), Execution time: mean = 39.162 us, total = 10.143 ms, Queueing time: mean = 95.225 us, max = 233.972 us, min = 8.714 us, total = 24.663 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 259 total (24 active), Execution time: mean = 57.949 s, total = 15008.700 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 211 total (0 active), Execution time: mean = 107.471 us, total = 22.676 ms, Queueing time: mean = 105.551 us, max = 190.637 us, min = 17.795 us, total = 22.271 ms InternalPubSubGcsService.grpc_server.GcsPublish - 211 total (0 active), Execution time: mean = 615.968 us, total = 129.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 488.179 us, total = 1.953 ms, Queueing time: mean = 67.763 us, max = 114.456 us, min = 54.388 us, total = 271.052 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:28:15,214 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3881 total (0 active) Queueing time: mean = 11.636 us, max = 1.999 ms, min = 563.000 ns, total = 45.160 ms Execution time: mean = 29.120 us, total = 113.014 ms Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 28.603 us, total = 109.863 ms, Queueing time: mean = 11.169 us, max = 1.999 ms, min = 3.185 us, total = 42.901 ms - 19 total (0 active), Execution time: mean = 135.583 us, total = 2.576 ms, Queueing time: mean = 104.191 us, max = 205.168 us, min = 15.988 us, total = 1.980 ms RaySyncer.BroadcastMessage - 19 total (0 active), Execution time: mean = 30.222 us, total = 574.216 us, Queueing time: mean = 949.316 ns, max = 1.380 us, min = 563.000 ns, total = 18.037 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:29:15,214 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 204 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 223 - cumulative published bytes: 177027 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 44 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 13 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 12 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:29:15,215 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 12622 total (4 active) Queueing time: mean = 209.688 us, max = 225.583 ms, min = 727.000 ns, total = 2.647 s Execution time: mean = 261.326 us, total = 3.298 s Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 24.991 us, total = 101.987 ms, Queueing time: mean = 11.163 us, max = 6.895 ms, min = 1.268 us, total = 45.556 ms GcsInMemoryStore.Put - 1679 total (0 active), Execution time: mean = 140.073 us, total = 235.183 ms, Queueing time: mean = 980.816 us, max = 225.278 ms, min = 1.495 us, total = 1.647 s RayletLoadPulled - 1020 total (1 active), Execution time: mean = 248.355 us, total = 253.322 ms, Queueing time: mean = 76.143 us, max = 1.351 ms, min = 10.621 us, total = 77.666 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1018 total (0 active), Execution time: mean = 75.410 us, total = 76.767 ms, Queueing time: mean = 99.134 us, max = 5.512 ms, min = 11.600 us, total = 100.918 ms NodeManagerService.grpc_client.GetResourceLoad - 1018 total (0 active), Execution time: mean = 1.443 ms, total = 1.469 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 423 total (0 active), Execution time: mean = 528.738 us, total = 223.656 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 423 total (0 active), Execution time: mean = 41.216 us, total = 17.434 ms, Queueing time: mean = 92.555 us, max = 239.072 us, min = 12.508 us, total = 39.151 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 408 total (0 active), Execution time: mean = 33.684 us, total = 13.743 ms, Queueing time: mean = 104.095 us, max = 277.841 us, min = 17.215 us, total = 42.471 ms NodeInfoGcsService.grpc_server.CheckAlive - 408 total (0 active), Execution time: mean = 539.020 us, total = 219.920 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 341 total (1 active), Execution time: mean = 10.281 us, total = 3.506 ms, Queueing time: mean = 73.571 us, max = 298.602 us, min = 19.990 us, total = 25.088 ms HealthCheck - 338 total (0 active), Execution time: mean = 7.727 us, total = 2.612 ms, Queueing time: mean = 88.357 us, max = 238.751 us, min = 12.322 us, total = 29.865 ms GcsInMemoryStore.Get - 264 total (0 active), Execution time: mean = 21.673 us, total = 5.722 ms, Queueing time: mean = 14.708 us, max = 137.899 us, min = 2.227 us, total = 3.883 ms InternalKVGcsService.grpc_server.InternalKVGet - 263 total (0 active), Execution time: mean = 827.727 us, total = 217.692 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 263 total (0 active), Execution time: mean = 53.269 us, total = 14.010 ms, Queueing time: mean = 356.745 us, max = 22.982 ms, min = 9.997 us, total = 93.824 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 204 total (0 active), Execution time: mean = 633.166 us, total = 129.166 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 204 total (0 active), Execution time: mean = 90.546 us, total = 18.471 ms, Queueing time: mean = 105.311 us, max = 225.322 us, min = 22.663 us, total = 21.483 ms GCSServer.deadline_timer.debug_state_dump - 102 total (1 active), Execution time: mean = 1.286 ms, total = 131.179 ms, Queueing time: mean = 65.675 us, max = 163.420 us, min = 14.502 us, total = 6.699 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsResourceManager::Update - 19 total (0 active), Execution time: mean = 109.540 us, total = 2.081 ms, Queueing time: mean = 94.057 us, max = 175.233 us, min = 20.234 us, total = 1.787 ms GCSServer.deadline_timer.debug_state_event_stats_print - 17 total (1 active, 1 running), Execution time: mean = 1.496 ms, total = 25.438 ms, Queueing time: mean = 54.555 us, max = 121.246 us, min = 18.282 us, total = 927.435 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:29:15,216 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 47043 total (1 active) Queueing time: mean = 46.704 us, max = 3.642 ms, min = 2.014 us, total = 2.197 s Execution time: mean = 242.535 us, total = 11.410 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 21378 total (0 active), Execution time: mean = 502.918 us, total = 10.751 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 21378 total (0 active), Execution time: mean = 26.009 us, total = 556.022 ms, Queueing time: mean = 100.110 us, max = 1.100 ms, min = 4.024 us, total = 2.140 s event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 24.500 us, total = 99.983 ms, Queueing time: mean = 8.775 us, max = 395.058 us, min = 2.014 us, total = 35.809 ms GcsTaskManager.GcJobSummary - 205 total (1 active), Execution time: mean = 10.566 us, total = 2.166 ms, Queueing time: mean = 103.125 us, max = 3.642 ms, min = 16.403 us, total = 21.141 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:29:15,216 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5124 total (25 active) Queueing time: mean = 18.158 us, max = 600.706 us, min = 2.413 us, total = 93.044 ms Execution time: mean = 2.941 s, total = 15069.117 s Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 27.606 us, total = 112.662 ms, Queueing time: mean = 10.077 us, max = 600.706 us, min = 2.413 us, total = 41.124 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 271 total (0 active), Execution time: mean = 39.497 us, total = 10.704 ms, Queueing time: mean = 96.074 us, max = 233.972 us, min = 8.714 us, total = 26.036 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 271 total (24 active), Execution time: mean = 55.604 s, total = 15068.795 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 223 total (0 active), Execution time: mean = 106.674 us, total = 23.788 ms, Queueing time: mean = 105.929 us, max = 190.637 us, min = 17.795 us, total = 23.622 ms InternalPubSubGcsService.grpc_server.GcsPublish - 223 total (0 active), Execution time: mean = 615.841 us, total = 137.333 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 488.179 us, total = 1.953 ms, Queueing time: mean = 67.763 us, max = 114.456 us, min = 54.388 us, total = 271.052 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:29:15,216 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4121 total (0 active) Queueing time: mean = 11.619 us, max = 1.999 ms, min = 563.000 ns, total = 47.881 ms Execution time: mean = 29.380 us, total = 121.077 ms Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 28.896 us, total = 117.926 ms, Queueing time: mean = 11.179 us, max = 1.999 ms, min = 3.173 us, total = 45.621 ms - 19 total (0 active), Execution time: mean = 135.583 us, total = 2.576 ms, Queueing time: mean = 104.191 us, max = 205.168 us, min = 15.988 us, total = 1.980 ms RaySyncer.BroadcastMessage - 19 total (0 active), Execution time: mean = 30.222 us, total = 574.216 us, Queueing time: mean = 949.316 ns, max = 1.380 us, min = 563.000 ns, total = 18.037 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:30:15,216 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 216 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 236 - cumulative published bytes: 187153 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:30:15,217 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 13343 total (4 active) Queueing time: mean = 200.104 us, max = 225.583 ms, min = 727.000 ns, total = 2.670 s Execution time: mean = 257.352 us, total = 3.434 s Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 24.888 us, total = 107.540 ms, Queueing time: mean = 11.045 us, max = 6.895 ms, min = 1.268 us, total = 47.725 ms GcsInMemoryStore.Put - 1775 total (0 active), Execution time: mean = 132.746 us, total = 235.623 ms, Queueing time: mean = 929.681 us, max = 225.278 ms, min = 1.495 us, total = 1.650 s RayletLoadPulled - 1080 total (1 active), Execution time: mean = 247.083 us, total = 266.850 ms, Queueing time: mean = 75.547 us, max = 1.351 ms, min = 10.621 us, total = 81.591 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1078 total (0 active), Execution time: mean = 74.996 us, total = 80.845 ms, Queueing time: mean = 97.926 us, max = 5.512 ms, min = 11.600 us, total = 105.564 ms NodeManagerService.grpc_client.GetResourceLoad - 1078 total (0 active), Execution time: mean = 1.427 ms, total = 1.538 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 447 total (0 active), Execution time: mean = 520.357 us, total = 232.600 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 447 total (0 active), Execution time: mean = 41.031 us, total = 18.341 ms, Queueing time: mean = 91.631 us, max = 239.072 us, min = 12.508 us, total = 40.959 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 432 total (0 active), Execution time: mean = 33.307 us, total = 14.389 ms, Queueing time: mean = 102.617 us, max = 277.841 us, min = 17.215 us, total = 44.330 ms NodeInfoGcsService.grpc_server.CheckAlive - 432 total (0 active), Execution time: mean = 530.133 us, total = 229.017 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 361 total (1 active), Execution time: mean = 10.181 us, total = 3.676 ms, Queueing time: mean = 72.676 us, max = 298.602 us, min = 19.990 us, total = 26.236 ms HealthCheck - 358 total (0 active), Execution time: mean = 7.693 us, total = 2.754 ms, Queueing time: mean = 87.660 us, max = 238.751 us, min = 12.322 us, total = 31.382 ms GcsInMemoryStore.Get - 276 total (0 active), Execution time: mean = 22.051 us, total = 6.086 ms, Queueing time: mean = 14.629 us, max = 137.899 us, min = 2.227 us, total = 4.038 ms InternalKVGcsService.grpc_server.InternalKVGet - 275 total (0 active), Execution time: mean = 810.561 us, total = 222.904 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 275 total (0 active), Execution time: mean = 54.161 us, total = 14.894 ms, Queueing time: mean = 344.580 us, max = 22.982 ms, min = 9.997 us, total = 94.759 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 216 total (0 active), Execution time: mean = 624.960 us, total = 134.991 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 216 total (0 active), Execution time: mean = 89.957 us, total = 19.431 ms, Queueing time: mean = 104.525 us, max = 225.322 us, min = 22.663 us, total = 22.577 ms GCSServer.deadline_timer.debug_state_dump - 108 total (1 active), Execution time: mean = 1.279 ms, total = 138.114 ms, Queueing time: mean = 65.297 us, max = 163.420 us, min = 14.502 us, total = 7.052 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 18 total (1 active, 1 running), Execution time: mean = 1.508 ms, total = 27.145 ms, Queueing time: mean = 54.270 us, max = 121.246 us, min = 18.282 us, total = 976.854 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:30:15,217 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 49815 total (1 active) Queueing time: mean = 45.384 us, max = 3.642 ms, min = 2.014 us, total = 2.261 s Execution time: mean = 237.040 us, total = 11.808 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 22638 total (0 active), Execution time: mean = 491.416 us, total = 11.125 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 22638 total (0 active), Execution time: mean = 25.430 us, total = 575.681 ms, Queueing time: mean = 97.239 us, max = 1.100 ms, min = 4.024 us, total = 2.201 s event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 24.423 us, total = 105.530 ms, Queueing time: mean = 8.731 us, max = 395.058 us, min = 2.014 us, total = 37.728 ms GcsTaskManager.GcJobSummary - 217 total (1 active), Execution time: mean = 10.416 us, total = 2.260 ms, Queueing time: mean = 100.173 us, max = 3.642 ms, min = 16.403 us, total = 21.738 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:30:15,217 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5416 total (25 active) Queueing time: mean = 17.843 us, max = 600.706 us, min = 2.413 us, total = 96.636 ms Execution time: mean = 2.793 s, total = 15129.345 s Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 27.577 us, total = 119.158 ms, Queueing time: mean = 10.044 us, max = 600.706 us, min = 2.413 us, total = 43.401 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 284 total (0 active), Execution time: mean = 39.485 us, total = 11.214 ms, Queueing time: mean = 93.705 us, max = 233.972 us, min = 8.714 us, total = 26.612 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 284 total (24 active), Execution time: mean = 53.271 s, total = 15129.009 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 236 total (0 active), Execution time: mean = 107.021 us, total = 25.257 ms, Queueing time: mean = 103.224 us, max = 190.637 us, min = 17.795 us, total = 24.361 ms InternalPubSubGcsService.grpc_server.GcsPublish - 236 total (0 active), Execution time: mean = 605.744 us, total = 142.956 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 488.179 us, total = 1.953 ms, Queueing time: mean = 67.763 us, max = 114.456 us, min = 54.388 us, total = 271.052 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:30:15,218 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4365 total (0 active) Queueing time: mean = 11.554 us, max = 1.999 ms, min = 563.000 ns, total = 50.432 ms Execution time: mean = 29.275 us, total = 127.788 ms Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 28.778 us, total = 124.351 ms, Queueing time: mean = 11.106 us, max = 1.999 ms, min = 3.173 us, total = 47.988 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:31:15,218 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 228 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 249 - cumulative published bytes: 197015 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:31:15,219 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 14062 total (4 active) Queueing time: mean = 191.728 us, max = 225.583 ms, min = 727.000 ns, total = 2.696 s Execution time: mean = 254.638 us, total = 3.581 s Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 24.732 us, total = 112.805 ms, Queueing time: mean = 10.938 us, max = 6.895 ms, min = 1.268 us, total = 49.886 ms GcsInMemoryStore.Put - 1871 total (0 active), Execution time: mean = 126.184 us, total = 236.090 ms, Queueing time: mean = 883.857 us, max = 225.278 ms, min = 1.495 us, total = 1.654 s RayletLoadPulled - 1140 total (1 active), Execution time: mean = 245.878 us, total = 280.301 ms, Queueing time: mean = 75.239 us, max = 1.351 ms, min = 10.621 us, total = 85.773 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1138 total (0 active), Execution time: mean = 75.036 us, total = 85.391 ms, Queueing time: mean = 97.635 us, max = 5.512 ms, min = 11.600 us, total = 111.109 ms NodeManagerService.grpc_client.GetResourceLoad - 1138 total (0 active), Execution time: mean = 1.418 ms, total = 1.614 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 471 total (0 active), Execution time: mean = 516.457 us, total = 243.251 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 471 total (0 active), Execution time: mean = 40.423 us, total = 19.039 ms, Queueing time: mean = 91.577 us, max = 239.072 us, min = 12.201 us, total = 43.133 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 456 total (0 active), Execution time: mean = 33.651 us, total = 15.345 ms, Queueing time: mean = 102.151 us, max = 277.841 us, min = 17.215 us, total = 46.581 ms NodeInfoGcsService.grpc_server.CheckAlive - 456 total (0 active), Execution time: mean = 527.025 us, total = 240.324 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 381 total (1 active), Execution time: mean = 10.122 us, total = 3.857 ms, Queueing time: mean = 73.718 us, max = 715.636 us, min = 19.990 us, total = 28.086 ms HealthCheck - 378 total (0 active), Execution time: mean = 7.670 us, total = 2.899 ms, Queueing time: mean = 87.729 us, max = 238.751 us, min = 12.322 us, total = 33.162 ms GcsInMemoryStore.Get - 288 total (0 active), Execution time: mean = 21.877 us, total = 6.301 ms, Queueing time: mean = 14.450 us, max = 137.899 us, min = 2.227 us, total = 4.162 ms InternalKVGcsService.grpc_server.InternalKVGet - 287 total (0 active), Execution time: mean = 796.350 us, total = 228.553 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 287 total (0 active), Execution time: mean = 53.948 us, total = 15.483 ms, Queueing time: mean = 333.057 us, max = 22.982 ms, min = 9.997 us, total = 95.587 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 228 total (0 active), Execution time: mean = 621.368 us, total = 141.672 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 228 total (0 active), Execution time: mean = 89.955 us, total = 20.510 ms, Queueing time: mean = 104.507 us, max = 225.322 us, min = 22.663 us, total = 23.828 ms GCSServer.deadline_timer.debug_state_dump - 114 total (1 active), Execution time: mean = 1.281 ms, total = 146.042 ms, Queueing time: mean = 65.192 us, max = 163.420 us, min = 14.502 us, total = 7.432 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 19 total (1 active, 1 running), Execution time: mean = 1.509 ms, total = 28.674 ms, Queueing time: mean = 54.022 us, max = 121.246 us, min = 18.282 us, total = 1.026 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:31:15,219 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 52587 total (1 active) Queueing time: mean = 44.960 us, max = 3.642 ms, min = 2.014 us, total = 2.364 s Execution time: mean = 235.180 us, total = 12.367 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 23898 total (0 active), Execution time: mean = 487.559 us, total = 11.652 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 23898 total (0 active), Execution time: mean = 25.209 us, total = 602.443 ms, Queueing time: mean = 96.332 us, max = 1.144 ms, min = 4.024 us, total = 2.302 s event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 24.317 us, total = 110.909 ms, Queueing time: mean = 8.697 us, max = 395.058 us, min = 2.014 us, total = 39.665 ms GcsTaskManager.GcJobSummary - 229 total (1 active), Execution time: mean = 10.312 us, total = 2.361 ms, Queueing time: mean = 98.187 us, max = 3.642 ms, min = 16.403 us, total = 22.485 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:31:15,219 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5708 total (25 active) Queueing time: mean = 17.649 us, max = 600.706 us, min = 2.413 us, total = 100.739 ms Execution time: mean = 2.661 s, total = 15189.492 s Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 27.370 us, total = 124.833 ms, Queueing time: mean = 9.982 us, max = 600.706 us, min = 2.413 us, total = 45.530 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 297 total (0 active), Execution time: mean = 39.337 us, total = 11.683 ms, Queueing time: mean = 92.668 us, max = 233.972 us, min = 8.714 us, total = 27.522 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 297 total (24 active), Execution time: mean = 51.142 s, total = 15189.142 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 249 total (0 active), Execution time: mean = 105.683 us, total = 26.315 ms, Queueing time: mean = 102.104 us, max = 190.637 us, min = 11.318 us, total = 25.424 ms InternalPubSubGcsService.grpc_server.GcsPublish - 249 total (0 active), Execution time: mean = 599.766 us, total = 149.342 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 488.179 us, total = 1.953 ms, Queueing time: mean = 67.763 us, max = 114.456 us, min = 54.388 us, total = 271.052 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:31:15,219 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4605 total (0 active) Queueing time: mean = 11.473 us, max = 1.999 ms, min = 563.000 ns, total = 52.831 ms Execution time: mean = 29.068 us, total = 133.857 ms Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 28.595 us, total = 130.421 ms, Queueing time: mean = 11.047 us, max = 1.999 ms, min = 3.173 us, total = 50.387 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:32:15,219 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 240 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 261 - cumulative published bytes: 206843 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:32:15,220 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 14780 total (4 active) Queueing time: mean = 184.508 us, max = 225.583 ms, min = 727.000 ns, total = 2.727 s Execution time: mean = 253.651 us, total = 3.749 s Event stats: event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 24.716 us, total = 118.638 ms, Queueing time: mean = 10.861 us, max = 6.895 ms, min = 1.268 us, total = 52.135 ms GcsInMemoryStore.Put - 1967 total (0 active), Execution time: mean = 120.311 us, total = 236.653 ms, Queueing time: mean = 843.656 us, max = 225.278 ms, min = 1.495 us, total = 1.659 s RayletLoadPulled - 1200 total (1 active), Execution time: mean = 245.990 us, total = 295.187 ms, Queueing time: mean = 75.403 us, max = 1.351 ms, min = 10.621 us, total = 90.484 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1198 total (0 active), Execution time: mean = 75.097 us, total = 89.966 ms, Queueing time: mean = 98.099 us, max = 5.512 ms, min = 11.600 us, total = 117.523 ms NodeManagerService.grpc_client.GetResourceLoad - 1198 total (0 active), Execution time: mean = 1.420 ms, total = 1.701 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 495 total (0 active), Execution time: mean = 517.174 us, total = 256.001 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 495 total (0 active), Execution time: mean = 40.134 us, total = 19.866 ms, Queueing time: mean = 92.134 us, max = 239.072 us, min = 12.201 us, total = 45.606 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 480 total (0 active), Execution time: mean = 34.046 us, total = 16.342 ms, Queueing time: mean = 102.751 us, max = 283.661 us, min = 17.215 us, total = 49.321 ms NodeInfoGcsService.grpc_server.CheckAlive - 480 total (0 active), Execution time: mean = 529.473 us, total = 254.147 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 401 total (1 active), Execution time: mean = 10.165 us, total = 4.076 ms, Queueing time: mean = 73.725 us, max = 715.636 us, min = 19.990 us, total = 29.564 ms HealthCheck - 398 total (0 active), Execution time: mean = 7.780 us, total = 3.097 ms, Queueing time: mean = 87.940 us, max = 238.751 us, min = 12.322 us, total = 35.000 ms GcsInMemoryStore.Get - 300 total (0 active), Execution time: mean = 22.087 us, total = 6.626 ms, Queueing time: mean = 14.371 us, max = 137.899 us, min = 2.227 us, total = 4.311 ms InternalKVGcsService.grpc_server.InternalKVGet - 299 total (0 active), Execution time: mean = 787.064 us, total = 235.332 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 299 total (0 active), Execution time: mean = 53.976 us, total = 16.139 ms, Queueing time: mean = 323.701 us, max = 22.982 ms, min = 9.997 us, total = 96.786 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 240 total (0 active), Execution time: mean = 621.289 us, total = 149.109 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 240 total (0 active), Execution time: mean = 89.570 us, total = 21.497 ms, Queueing time: mean = 105.203 us, max = 225.322 us, min = 22.663 us, total = 25.249 ms GCSServer.deadline_timer.debug_state_dump - 120 total (1 active), Execution time: mean = 1.288 ms, total = 154.545 ms, Queueing time: mean = 65.766 us, max = 163.420 us, min = 14.502 us, total = 7.892 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 20 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 30.284 ms, Queueing time: mean = 53.700 us, max = 121.246 us, min = 18.282 us, total = 1.074 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:32:15,220 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 55358 total (1 active) Queueing time: mean = 45.190 us, max = 3.642 ms, min = 2.014 us, total = 2.502 s Execution time: mean = 235.456 us, total = 13.034 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 25158 total (0 active), Execution time: mean = 488.192 us, total = 12.282 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 25158 total (0 active), Execution time: mean = 25.171 us, total = 633.251 ms, Queueing time: mean = 96.856 us, max = 1.144 ms, min = 4.024 us, total = 2.437 s event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 24.317 us, total = 116.720 ms, Queueing time: mean = 8.684 us, max = 395.058 us, min = 2.014 us, total = 41.682 ms GcsTaskManager.GcJobSummary - 241 total (1 active), Execution time: mean = 10.238 us, total = 2.467 ms, Queueing time: mean = 96.433 us, max = 3.642 ms, min = 16.403 us, total = 23.240 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:32:15,220 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6040 total (25 active) Queueing time: mean = 17.667 us, max = 600.706 us, min = 2.413 us, total = 106.712 ms Execution time: mean = 4.710 s, total = 28449.700 s Event stats: event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 27.158 us, total = 130.361 ms, Queueing time: mean = 9.928 us, max = 600.706 us, min = 2.413 us, total = 47.653 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 331 total (0 active), Execution time: mean = 38.140 us, total = 12.624 ms, Queueing time: mean = 91.725 us, max = 233.972 us, min = 8.714 us, total = 30.361 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 331 total (24 active), Execution time: mean = 85.950 s, total = 28449.333 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 261 total (0 active), Execution time: mean = 105.164 us, total = 27.448 ms, Queueing time: mean = 100.947 us, max = 190.637 us, min = 11.318 us, total = 26.347 ms InternalPubSubGcsService.grpc_server.GcsPublish - 261 total (0 active), Execution time: mean = 600.105 us, total = 156.627 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 678.110 us, total = 3.391 ms, Queueing time: mean = 71.756 us, max = 114.456 us, min = 54.388 us, total = 358.779 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:32:15,220 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4844 total (0 active) Queueing time: mean = 11.462 us, max = 1.999 ms, min = 563.000 ns, total = 55.520 ms Execution time: mean = 29.113 us, total = 141.021 ms Event stats: event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 28.664 us, total = 137.585 ms, Queueing time: mean = 11.058 us, max = 1.999 ms, min = 3.133 us, total = 53.077 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:33:15,221 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 252 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 273 - cumulative published bytes: 216671 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:33:15,222 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 15499 total (4 active) Queueing time: mean = 177.802 us, max = 225.583 ms, min = 727.000 ns, total = 2.756 s Execution time: mean = 252.787 us, total = 3.918 s Event stats: event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 24.733 us, total = 124.656 ms, Queueing time: mean = 10.791 us, max = 6.895 ms, min = 1.268 us, total = 54.385 ms GcsInMemoryStore.Put - 2063 total (0 active), Execution time: mean = 114.943 us, total = 237.127 ms, Queueing time: mean = 806.459 us, max = 225.278 ms, min = 1.495 us, total = 1.664 s RayletLoadPulled - 1260 total (1 active), Execution time: mean = 245.882 us, total = 309.811 ms, Queueing time: mean = 75.449 us, max = 1.351 ms, min = 10.621 us, total = 95.065 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1258 total (0 active), Execution time: mean = 75.030 us, total = 94.388 ms, Queueing time: mean = 98.350 us, max = 5.512 ms, min = 11.600 us, total = 123.725 ms NodeManagerService.grpc_client.GetResourceLoad - 1258 total (0 active), Execution time: mean = 1.423 ms, total = 1.790 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 519 total (0 active), Execution time: mean = 519.088 us, total = 269.407 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 519 total (0 active), Execution time: mean = 40.420 us, total = 20.978 ms, Queueing time: mean = 91.892 us, max = 239.072 us, min = 12.201 us, total = 47.692 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 504 total (0 active), Execution time: mean = 34.078 us, total = 17.175 ms, Queueing time: mean = 103.555 us, max = 283.661 us, min = 17.215 us, total = 52.192 ms NodeInfoGcsService.grpc_server.CheckAlive - 504 total (0 active), Execution time: mean = 532.085 us, total = 268.171 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 421 total (1 active), Execution time: mean = 10.219 us, total = 4.302 ms, Queueing time: mean = 73.919 us, max = 715.636 us, min = 19.990 us, total = 31.120 ms HealthCheck - 418 total (0 active), Execution time: mean = 7.783 us, total = 3.253 ms, Queueing time: mean = 88.130 us, max = 238.751 us, min = 11.335 us, total = 36.838 ms GcsInMemoryStore.Get - 312 total (0 active), Execution time: mean = 21.898 us, total = 6.832 ms, Queueing time: mean = 14.232 us, max = 137.899 us, min = 2.227 us, total = 4.441 ms InternalKVGcsService.grpc_server.InternalKVGet - 311 total (0 active), Execution time: mean = 778.094 us, total = 241.987 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 311 total (0 active), Execution time: mean = 53.830 us, total = 16.741 ms, Queueing time: mean = 314.818 us, max = 22.982 ms, min = 9.997 us, total = 97.908 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 252 total (0 active), Execution time: mean = 624.071 us, total = 157.266 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 252 total (0 active), Execution time: mean = 89.119 us, total = 22.458 ms, Queueing time: mean = 105.624 us, max = 225.322 us, min = 22.663 us, total = 26.617 ms GCSServer.deadline_timer.debug_state_dump - 126 total (1 active), Execution time: mean = 1.284 ms, total = 161.841 ms, Queueing time: mean = 65.871 us, max = 163.420 us, min = 14.502 us, total = 8.300 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GCSServer.deadline_timer.debug_state_event_stats_print - 21 total (1 active, 1 running), Execution time: mean = 1.502 ms, total = 31.532 ms, Queueing time: mean = 53.948 us, max = 121.246 us, min = 18.282 us, total = 1.133 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:33:15,222 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 58130 total (1 active) Queueing time: mean = 45.492 us, max = 3.642 ms, min = 2.014 us, total = 2.644 s Execution time: mean = 236.366 us, total = 13.740 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 26418 total (0 active), Execution time: mean = 490.137 us, total = 12.948 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 26418 total (0 active), Execution time: mean = 25.215 us, total = 666.140 ms, Queueing time: mean = 97.529 us, max = 1.144 ms, min = 4.024 us, total = 2.577 s event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 24.359 us, total = 122.767 ms, Queueing time: mean = 8.688 us, max = 395.058 us, min = 2.014 us, total = 43.787 ms GcsTaskManager.GcJobSummary - 253 total (1 active), Execution time: mean = 10.302 us, total = 2.606 ms, Queueing time: mean = 95.370 us, max = 3.642 ms, min = 16.403 us, total = 24.129 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:33:15,222 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6328 total (25 active) Queueing time: mean = 17.705 us, max = 600.706 us, min = 2.413 us, total = 112.035 ms Execution time: mean = 4.505 s, total = 28509.826 s Event stats: event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 27.196 us, total = 137.067 ms, Queueing time: mean = 9.932 us, max = 600.706 us, min = 2.413 us, total = 50.059 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 343 total (0 active), Execution time: mean = 38.531 us, total = 13.216 ms, Queueing time: mean = 92.691 us, max = 233.972 us, min = 8.714 us, total = 31.793 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 343 total (24 active), Execution time: mean = 83.118 s, total = 28509.443 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 273 total (0 active), Execution time: mean = 105.762 us, total = 28.873 ms, Queueing time: mean = 101.955 us, max = 190.637 us, min = 11.318 us, total = 27.834 ms InternalPubSubGcsService.grpc_server.GcsPublish - 273 total (0 active), Execution time: mean = 601.835 us, total = 164.301 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 678.110 us, total = 3.391 ms, Queueing time: mean = 71.756 us, max = 114.456 us, min = 54.388 us, total = 358.779 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:33:15,223 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5084 total (0 active) Queueing time: mean = 11.452 us, max = 1.999 ms, min = 563.000 ns, total = 58.222 ms Execution time: mean = 29.124 us, total = 148.065 ms Event stats: event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 28.696 us, total = 144.629 ms, Queueing time: mean = 11.067 us, max = 1.999 ms, min = 3.133 us, total = 55.779 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:34:15,223 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 264 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 285 - cumulative published bytes: 226499 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:34:15,224 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 16218 total (4 active) Queueing time: mean = 171.496 us, max = 225.583 ms, min = 727.000 ns, total = 2.781 s Execution time: mean = 251.707 us, total = 4.082 s Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 24.731 us, total = 130.579 ms, Queueing time: mean = 10.731 us, max = 6.895 ms, min = 1.268 us, total = 56.658 ms GcsInMemoryStore.Put - 2159 total (0 active), Execution time: mean = 110.028 us, total = 237.551 ms, Queueing time: mean = 772.190 us, max = 225.278 ms, min = 1.495 us, total = 1.667 s RayletLoadPulled - 1320 total (1 active), Execution time: mean = 244.952 us, total = 323.337 ms, Queueing time: mean = 75.109 us, max = 1.351 ms, min = 10.621 us, total = 99.143 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1318 total (0 active), Execution time: mean = 74.712 us, total = 98.471 ms, Queueing time: mean = 98.022 us, max = 5.512 ms, min = 11.600 us, total = 129.193 ms NodeManagerService.grpc_client.GetResourceLoad - 1318 total (0 active), Execution time: mean = 1.424 ms, total = 1.877 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 543 total (0 active), Execution time: mean = 519.619 us, total = 282.153 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 543 total (0 active), Execution time: mean = 40.033 us, total = 21.738 ms, Queueing time: mean = 91.903 us, max = 239.072 us, min = 12.201 us, total = 49.904 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 528 total (0 active), Execution time: mean = 33.895 us, total = 17.896 ms, Queueing time: mean = 103.383 us, max = 283.661 us, min = 17.215 us, total = 54.586 ms NodeInfoGcsService.grpc_server.CheckAlive - 528 total (0 active), Execution time: mean = 532.078 us, total = 280.937 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 441 total (1 active), Execution time: mean = 10.294 us, total = 4.540 ms, Queueing time: mean = 73.631 us, max = 715.636 us, min = 19.990 us, total = 32.471 ms HealthCheck - 438 total (0 active), Execution time: mean = 7.772 us, total = 3.404 ms, Queueing time: mean = 87.664 us, max = 238.751 us, min = 11.335 us, total = 38.397 ms GcsInMemoryStore.Get - 324 total (0 active), Execution time: mean = 21.729 us, total = 7.040 ms, Queueing time: mean = 14.094 us, max = 137.899 us, min = 2.227 us, total = 4.567 ms InternalKVGcsService.grpc_server.InternalKVGet - 323 total (0 active), Execution time: mean = 769.225 us, total = 248.460 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 323 total (0 active), Execution time: mean = 53.790 us, total = 17.374 ms, Queueing time: mean = 306.134 us, max = 22.982 ms, min = 9.997 us, total = 98.881 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 264 total (0 active), Execution time: mean = 624.219 us, total = 164.794 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 264 total (0 active), Execution time: mean = 88.872 us, total = 23.462 ms, Queueing time: mean = 105.291 us, max = 225.322 us, min = 22.663 us, total = 27.797 ms GCSServer.deadline_timer.debug_state_dump - 132 total (1 active), Execution time: mean = 1.284 ms, total = 169.443 ms, Queueing time: mean = 65.799 us, max = 163.420 us, min = 14.502 us, total = 8.685 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 22 total (1 active, 1 running), Execution time: mean = 1.520 ms, total = 33.431 ms, Queueing time: mean = 57.756 us, max = 137.733 us, min = 18.282 us, total = 1.271 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:34:15,224 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 60902 total (1 active) Queueing time: mean = 45.750 us, max = 3.642 ms, min = 2.014 us, total = 2.786 s Execution time: mean = 237.133 us, total = 14.442 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 27678 total (0 active), Execution time: mean = 491.807 us, total = 13.612 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 27678 total (0 active), Execution time: mean = 25.222 us, total = 698.105 ms, Queueing time: mean = 98.100 us, max = 1.144 ms, min = 4.024 us, total = 2.715 s event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 24.383 us, total = 128.743 ms, Queueing time: mean = 8.691 us, max = 395.058 us, min = 2.014 us, total = 45.891 ms GcsTaskManager.GcJobSummary - 265 total (1 active), Execution time: mean = 10.350 us, total = 2.743 ms, Queueing time: mean = 94.906 us, max = 3.642 ms, min = 16.403 us, total = 25.150 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:34:15,224 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6616 total (25 active) Queueing time: mean = 17.641 us, max = 600.706 us, min = 2.413 us, total = 116.715 ms Execution time: mean = 4.318 s, total = 28570.027 s Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 27.032 us, total = 142.730 ms, Queueing time: mean = 9.867 us, max = 600.706 us, min = 2.413 us, total = 52.098 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 355 total (0 active), Execution time: mean = 38.985 us, total = 13.840 ms, Queueing time: mean = 93.153 us, max = 233.972 us, min = 8.714 us, total = 33.069 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 355 total (24 active), Execution time: mean = 80.478 s, total = 28569.629 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 285 total (0 active), Execution time: mean = 105.497 us, total = 30.067 ms, Queueing time: mean = 102.446 us, max = 205.832 us, min = 11.318 us, total = 29.197 ms InternalPubSubGcsService.grpc_server.GcsPublish - 285 total (0 active), Execution time: mean = 601.661 us, total = 171.473 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 678.110 us, total = 3.391 ms, Queueing time: mean = 71.756 us, max = 114.456 us, min = 54.388 us, total = 358.779 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:34:15,224 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5324 total (0 active) Queueing time: mean = 11.458 us, max = 1.999 ms, min = 563.000 ns, total = 61.001 ms Execution time: mean = 29.144 us, total = 155.161 ms Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 28.736 us, total = 151.725 ms, Queueing time: mean = 11.090 us, max = 1.999 ms, min = 3.133 us, total = 58.558 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:35:15,224 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 276 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 298 - cumulative published bytes: 236361 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:35:15,225 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 16937 total (4 active) Queueing time: mean = 165.875 us, max = 225.583 ms, min = 727.000 ns, total = 2.809 s Execution time: mean = 250.611 us, total = 4.245 s Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 24.807 us, total = 136.932 ms, Queueing time: mean = 10.699 us, max = 6.895 ms, min = 1.268 us, total = 59.059 ms GcsInMemoryStore.Put - 2255 total (0 active), Execution time: mean = 105.557 us, total = 238.030 ms, Queueing time: mean = 741.109 us, max = 225.278 ms, min = 1.495 us, total = 1.671 s RayletLoadPulled - 1380 total (1 active), Execution time: mean = 244.921 us, total = 337.992 ms, Queueing time: mean = 74.988 us, max = 1.351 ms, min = 10.621 us, total = 103.484 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1378 total (0 active), Execution time: mean = 74.668 us, total = 102.892 ms, Queueing time: mean = 97.970 us, max = 5.512 ms, min = 11.600 us, total = 135.003 ms NodeManagerService.grpc_client.GetResourceLoad - 1378 total (0 active), Execution time: mean = 1.423 ms, total = 1.961 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 567 total (0 active), Execution time: mean = 519.978 us, total = 294.828 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 567 total (0 active), Execution time: mean = 39.923 us, total = 22.636 ms, Queueing time: mean = 92.201 us, max = 239.072 us, min = 12.201 us, total = 52.278 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 552 total (0 active), Execution time: mean = 34.310 us, total = 18.939 ms, Queueing time: mean = 104.264 us, max = 283.661 us, min = 17.215 us, total = 57.554 ms NodeInfoGcsService.grpc_server.CheckAlive - 552 total (0 active), Execution time: mean = 531.270 us, total = 293.261 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 461 total (1 active), Execution time: mean = 10.304 us, total = 4.750 ms, Queueing time: mean = 73.258 us, max = 715.636 us, min = 19.990 us, total = 33.772 ms HealthCheck - 458 total (0 active), Execution time: mean = 7.825 us, total = 3.584 ms, Queueing time: mean = 87.791 us, max = 238.751 us, min = 11.335 us, total = 40.208 ms GcsInMemoryStore.Get - 336 total (0 active), Execution time: mean = 21.736 us, total = 7.303 ms, Queueing time: mean = 13.998 us, max = 137.899 us, min = 2.227 us, total = 4.703 ms InternalKVGcsService.grpc_server.InternalKVGet - 335 total (0 active), Execution time: mean = 759.084 us, total = 254.293 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 335 total (0 active), Execution time: mean = 53.861 us, total = 18.044 ms, Queueing time: mean = 298.575 us, max = 22.982 ms, min = 9.997 us, total = 100.023 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 276 total (0 active), Execution time: mean = 624.232 us, total = 172.288 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 276 total (0 active), Execution time: mean = 89.205 us, total = 24.621 ms, Queueing time: mean = 105.326 us, max = 225.322 us, min = 22.663 us, total = 29.070 ms GCSServer.deadline_timer.debug_state_dump - 138 total (1 active), Execution time: mean = 1.290 ms, total = 178.082 ms, Queueing time: mean = 66.010 us, max = 163.420 us, min = 14.502 us, total = 9.109 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 23 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 35.086 ms, Queueing time: mean = 58.144 us, max = 137.733 us, min = 18.282 us, total = 1.337 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:35:15,225 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 63674 total (1 active) Queueing time: mean = 45.760 us, max = 3.642 ms, min = 2.014 us, total = 2.914 s Execution time: mean = 236.855 us, total = 15.082 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 28938 total (0 active), Execution time: mean = 491.268 us, total = 14.216 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 28938 total (0 active), Execution time: mean = 25.145 us, total = 727.639 ms, Queueing time: mean = 98.129 us, max = 1.144 ms, min = 4.024 us, total = 2.840 s event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 24.399 us, total = 134.684 ms, Queueing time: mean = 8.685 us, max = 395.058 us, min = 2.014 us, total = 47.939 ms GcsTaskManager.GcJobSummary - 277 total (1 active), Execution time: mean = 10.351 us, total = 2.867 ms, Queueing time: mean = 94.181 us, max = 3.642 ms, min = 16.403 us, total = 26.088 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:35:15,225 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6908 total (25 active) Queueing time: mean = 17.562 us, max = 600.706 us, min = 2.413 us, total = 121.319 ms Execution time: mean = 4.144 s, total = 28630.115 s Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 26.970 us, total = 148.873 ms, Queueing time: mean = 9.851 us, max = 600.706 us, min = 2.413 us, total = 54.379 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 368 total (0 active), Execution time: mean = 39.106 us, total = 14.391 ms, Queueing time: mean = 92.545 us, max = 233.972 us, min = 8.714 us, total = 34.056 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 368 total (24 active), Execution time: mean = 77.798 s, total = 28629.701 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 298 total (0 active), Execution time: mean = 105.456 us, total = 31.426 ms, Queueing time: mean = 102.462 us, max = 205.832 us, min = 11.318 us, total = 30.534 ms InternalPubSubGcsService.grpc_server.GcsPublish - 298 total (0 active), Execution time: mean = 602.002 us, total = 179.397 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 678.110 us, total = 3.391 ms, Queueing time: mean = 71.756 us, max = 114.456 us, min = 54.388 us, total = 358.779 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:35:15,225 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5564 total (0 active) Queueing time: mean = 11.883 us, max = 2.505 ms, min = 563.000 ns, total = 66.118 ms Execution time: mean = 29.127 us, total = 162.062 ms Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 28.737 us, total = 158.626 ms, Queueing time: mean = 11.535 us, max = 2.505 ms, min = 3.133 us, total = 63.675 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:36:15,226 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 288 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 310 - cumulative published bytes: 246189 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:36:15,227 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 17656 total (4 active) Queueing time: mean = 160.690 us, max = 225.583 ms, min = 727.000 ns, total = 2.837 s Execution time: mean = 250.176 us, total = 4.417 s Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 24.899 us, total = 143.418 ms, Queueing time: mean = 10.654 us, max = 6.895 ms, min = 1.268 us, total = 61.367 ms GcsInMemoryStore.Put - 2351 total (0 active), Execution time: mean = 101.452 us, total = 238.513 ms, Queueing time: mean = 712.629 us, max = 225.278 ms, min = 1.495 us, total = 1.675 s RayletLoadPulled - 1440 total (1 active), Execution time: mean = 245.149 us, total = 353.015 ms, Queueing time: mean = 74.902 us, max = 1.351 ms, min = 10.621 us, total = 107.859 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1438 total (0 active), Execution time: mean = 74.749 us, total = 107.489 ms, Queueing time: mean = 97.869 us, max = 5.512 ms, min = 11.600 us, total = 140.736 ms NodeManagerService.grpc_client.GetResourceLoad - 1438 total (0 active), Execution time: mean = 1.427 ms, total = 2.052 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 591 total (0 active), Execution time: mean = 521.881 us, total = 308.432 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 591 total (0 active), Execution time: mean = 40.215 us, total = 23.767 ms, Queueing time: mean = 93.352 us, max = 601.063 us, min = 12.201 us, total = 55.171 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 576 total (0 active), Execution time: mean = 34.340 us, total = 19.780 ms, Queueing time: mean = 104.113 us, max = 283.661 us, min = 17.215 us, total = 59.969 ms NodeInfoGcsService.grpc_server.CheckAlive - 576 total (0 active), Execution time: mean = 530.989 us, total = 305.850 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 481 total (1 active), Execution time: mean = 10.300 us, total = 4.954 ms, Queueing time: mean = 73.069 us, max = 715.636 us, min = 19.990 us, total = 35.146 ms HealthCheck - 478 total (0 active), Execution time: mean = 7.790 us, total = 3.724 ms, Queueing time: mean = 87.576 us, max = 238.751 us, min = 11.335 us, total = 41.861 ms GcsInMemoryStore.Get - 348 total (0 active), Execution time: mean = 22.151 us, total = 7.709 ms, Queueing time: mean = 13.898 us, max = 137.899 us, min = 2.227 us, total = 4.836 ms InternalKVGcsService.grpc_server.InternalKVGet - 347 total (0 active), Execution time: mean = 752.955 us, total = 261.275 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 347 total (0 active), Execution time: mean = 54.378 us, total = 18.869 ms, Queueing time: mean = 291.130 us, max = 22.982 ms, min = 9.997 us, total = 101.022 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 288 total (0 active), Execution time: mean = 624.047 us, total = 179.726 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 288 total (0 active), Execution time: mean = 89.185 us, total = 25.685 ms, Queueing time: mean = 105.474 us, max = 225.322 us, min = 22.663 us, total = 30.376 ms GCSServer.deadline_timer.debug_state_dump - 144 total (1 active), Execution time: mean = 1.292 ms, total = 186.002 ms, Queueing time: mean = 65.559 us, max = 163.420 us, min = 14.502 us, total = 9.440 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 24 total (1 active, 1 running), Execution time: mean = 1.504 ms, total = 36.102 ms, Queueing time: mean = 56.346 us, max = 137.733 us, min = 14.993 us, total = 1.352 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:36:15,227 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 66446 total (1 active) Queueing time: mean = 46.036 us, max = 3.642 ms, min = 2.014 us, total = 3.059 s Execution time: mean = 237.629 us, total = 15.789 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 30198 total (0 active), Execution time: mean = 492.859 us, total = 14.883 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 30198 total (0 active), Execution time: mean = 25.251 us, total = 762.529 ms, Queueing time: mean = 98.741 us, max = 1.144 ms, min = 4.024 us, total = 2.982 s event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 24.409 us, total = 140.598 ms, Queueing time: mean = 8.679 us, max = 395.058 us, min = 2.014 us, total = 49.989 ms GcsTaskManager.GcJobSummary - 289 total (1 active), Execution time: mean = 10.391 us, total = 3.003 ms, Queueing time: mean = 93.699 us, max = 3.642 ms, min = 16.403 us, total = 27.079 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:36:15,227 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7196 total (25 active) Queueing time: mean = 17.561 us, max = 600.706 us, min = 2.413 us, total = 126.367 ms Execution time: mean = 3.987 s, total = 28690.380 s Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 27.008 us, total = 155.565 ms, Queueing time: mean = 9.855 us, max = 600.706 us, min = 2.413 us, total = 56.766 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 380 total (0 active), Execution time: mean = 39.491 us, total = 15.006 ms, Queueing time: mean = 93.179 us, max = 233.972 us, min = 8.714 us, total = 35.408 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 380 total (24 active), Execution time: mean = 75.500 s, total = 28689.950 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 310 total (0 active), Execution time: mean = 105.331 us, total = 32.653 ms, Queueing time: mean = 102.717 us, max = 205.832 us, min = 11.318 us, total = 31.842 ms InternalPubSubGcsService.grpc_server.GcsPublish - 310 total (0 active), Execution time: mean = 602.466 us, total = 186.765 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 678.110 us, total = 3.391 ms, Queueing time: mean = 71.756 us, max = 114.456 us, min = 54.388 us, total = 358.779 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:36:15,227 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5804 total (0 active) Queueing time: mean = 11.820 us, max = 2.505 ms, min = 563.000 ns, total = 68.602 ms Execution time: mean = 29.041 us, total = 168.555 ms Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 28.667 us, total = 165.119 ms, Queueing time: mean = 11.486 us, max = 2.505 ms, min = 3.133 us, total = 66.158 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:37:15,228 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 300 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 322 - cumulative published bytes: 256017 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 51 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:37:15,228 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 18375 total (4 active) Queueing time: mean = 155.961 us, max = 225.583 ms, min = 727.000 ns, total = 2.866 s Execution time: mean = 249.793 us, total = 4.590 s Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 24.857 us, total = 149.144 ms, Queueing time: mean = 10.603 us, max = 6.895 ms, min = 1.268 us, total = 63.616 ms GcsInMemoryStore.Put - 2447 total (0 active), Execution time: mean = 97.689 us, total = 239.044 ms, Queueing time: mean = 686.677 us, max = 225.278 ms, min = 1.495 us, total = 1.680 s RayletLoadPulled - 1500 total (1 active), Execution time: mean = 245.790 us, total = 368.684 ms, Queueing time: mean = 74.921 us, max = 1.351 ms, min = 10.621 us, total = 112.381 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1498 total (0 active), Execution time: mean = 74.979 us, total = 112.319 ms, Queueing time: mean = 97.747 us, max = 5.512 ms, min = 11.600 us, total = 146.425 ms NodeManagerService.grpc_client.GetResourceLoad - 1498 total (0 active), Execution time: mean = 1.432 ms, total = 2.145 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 615 total (0 active), Execution time: mean = 523.243 us, total = 321.794 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 615 total (0 active), Execution time: mean = 40.039 us, total = 24.624 ms, Queueing time: mean = 93.383 us, max = 601.063 us, min = 12.201 us, total = 57.430 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 600 total (0 active), Execution time: mean = 34.210 us, total = 20.526 ms, Queueing time: mean = 104.465 us, max = 283.661 us, min = 17.215 us, total = 62.679 ms NodeInfoGcsService.grpc_server.CheckAlive - 600 total (0 active), Execution time: mean = 530.949 us, total = 318.569 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 501 total (1 active), Execution time: mean = 10.306 us, total = 5.163 ms, Queueing time: mean = 72.882 us, max = 715.636 us, min = 19.990 us, total = 36.514 ms HealthCheck - 498 total (0 active), Execution time: mean = 7.801 us, total = 3.885 ms, Queueing time: mean = 88.120 us, max = 238.751 us, min = 11.335 us, total = 43.884 ms GcsInMemoryStore.Get - 360 total (0 active), Execution time: mean = 22.092 us, total = 7.953 ms, Queueing time: mean = 13.810 us, max = 137.899 us, min = 2.227 us, total = 4.972 ms InternalKVGcsService.grpc_server.InternalKVGet - 359 total (0 active), Execution time: mean = 747.466 us, total = 268.340 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 359 total (0 active), Execution time: mean = 54.221 us, total = 19.465 ms, Queueing time: mean = 284.552 us, max = 22.982 ms, min = 9.997 us, total = 102.154 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 300 total (0 active), Execution time: mean = 623.634 us, total = 187.090 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 300 total (0 active), Execution time: mean = 88.964 us, total = 26.689 ms, Queueing time: mean = 105.350 us, max = 225.322 us, min = 22.663 us, total = 31.605 ms GCSServer.deadline_timer.debug_state_dump - 150 total (1 active), Execution time: mean = 1.289 ms, total = 193.421 ms, Queueing time: mean = 65.233 us, max = 163.420 us, min = 14.502 us, total = 9.785 ms GCSServer.deadline_timer.debug_state_event_stats_print - 25 total (1 active, 1 running), Execution time: mean = 1.523 ms, total = 38.063 ms, Queueing time: mean = 56.778 us, max = 137.733 us, min = 14.993 us, total = 1.419 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 21 total (0 active), Execution time: mean = 122.810 us, total = 2.579 ms, Queueing time: mean = 97.276 us, max = 175.233 us, min = 20.234 us, total = 2.043 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:37:15,228 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 69218 total (1 active) Queueing time: mean = 46.110 us, max = 3.642 ms, min = 2.014 us, total = 3.192 s Execution time: mean = 237.827 us, total = 16.462 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 31458 total (0 active), Execution time: mean = 493.257 us, total = 15.517 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 31458 total (0 active), Execution time: mean = 25.283 us, total = 795.364 ms, Queueing time: mean = 98.909 us, max = 1.144 ms, min = 4.024 us, total = 3.111 s event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 24.424 us, total = 146.547 ms, Queueing time: mean = 8.685 us, max = 395.058 us, min = 2.014 us, total = 52.108 ms GcsTaskManager.GcJobSummary - 301 total (1 active), Execution time: mean = 10.381 us, total = 3.125 ms, Queueing time: mean = 93.048 us, max = 3.642 ms, min = 16.403 us, total = 28.008 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:37:15,228 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7487 total (25 active) Queueing time: mean = 17.555 us, max = 600.706 us, min = 2.413 us, total = 131.433 ms Execution time: mean = 3.920 s, total = 29350.463 s Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 27.041 us, total = 162.245 ms, Queueing time: mean = 9.863 us, max = 600.706 us, min = 2.413 us, total = 59.180 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 393 total (0 active), Execution time: mean = 39.494 us, total = 15.521 ms, Queueing time: mean = 93.432 us, max = 233.972 us, min = 8.714 us, total = 36.719 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 393 total (24 active), Execution time: mean = 74.682 s, total = 29350.017 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 322 total (0 active), Execution time: mean = 105.151 us, total = 33.859 ms, Queueing time: mean = 102.811 us, max = 205.832 us, min = 11.318 us, total = 33.105 ms InternalPubSubGcsService.grpc_server.GcsPublish - 322 total (0 active), Execution time: mean = 603.865 us, total = 194.444 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 572.887 us, total = 3.437 ms, Queueing time: mean = 72.940 us, max = 114.456 us, min = 54.388 us, total = 437.639 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:37:15,228 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6044 total (0 active) Queueing time: mean = 11.793 us, max = 2.505 ms, min = 563.000 ns, total = 71.280 ms Execution time: mean = 29.062 us, total = 175.651 ms Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 28.702 us, total = 172.215 ms, Queueing time: mean = 11.473 us, max = 2.505 ms, min = 3.133 us, total = 68.836 ms - 21 total (0 active), Execution time: mean = 132.794 us, total = 2.789 ms, Queueing time: mean = 102.925 us, max = 205.168 us, min = 15.988 us, total = 2.161 ms RaySyncer.BroadcastMessage - 21 total (0 active), Execution time: mean = 30.779 us, total = 646.365 us, Queueing time: mean = 960.381 ns, max = 1.380 us, min = 563.000 ns, total = 20.168 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:38:15,228 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 312 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 335 - cumulative published bytes: 266146 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:38:15,229 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 19095 total (4 active) Queueing time: mean = 151.728 us, max = 225.583 ms, min = 727.000 ns, total = 2.897 s Execution time: mean = 249.665 us, total = 4.767 s Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 24.954 us, total = 155.713 ms, Queueing time: mean = 10.593 us, max = 6.895 ms, min = 1.268 us, total = 66.098 ms GcsInMemoryStore.Put - 2543 total (0 active), Execution time: mean = 94.215 us, total = 239.589 ms, Queueing time: mean = 662.416 us, max = 225.278 ms, min = 1.495 us, total = 1.685 s RayletLoadPulled - 1560 total (1 active), Execution time: mean = 246.245 us, total = 384.142 ms, Queueing time: mean = 75.131 us, max = 1.351 ms, min = 10.621 us, total = 117.204 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1558 total (0 active), Execution time: mean = 75.363 us, total = 117.416 ms, Queueing time: mean = 98.356 us, max = 5.512 ms, min = 11.600 us, total = 153.239 ms NodeManagerService.grpc_client.GetResourceLoad - 1558 total (0 active), Execution time: mean = 1.437 ms, total = 2.238 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 639 total (0 active), Execution time: mean = 523.799 us, total = 334.708 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 639 total (0 active), Execution time: mean = 40.093 us, total = 25.619 ms, Queueing time: mean = 93.998 us, max = 601.063 us, min = 12.201 us, total = 60.065 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 624 total (0 active), Execution time: mean = 34.966 us, total = 21.819 ms, Queueing time: mean = 105.292 us, max = 283.661 us, min = 17.215 us, total = 65.702 ms NodeInfoGcsService.grpc_server.CheckAlive - 624 total (0 active), Execution time: mean = 533.713 us, total = 333.037 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 521 total (1 active), Execution time: mean = 10.739 us, total = 5.595 ms, Queueing time: mean = 73.220 us, max = 715.636 us, min = 19.990 us, total = 38.147 ms HealthCheck - 518 total (0 active), Execution time: mean = 7.844 us, total = 4.063 ms, Queueing time: mean = 88.687 us, max = 238.751 us, min = 11.335 us, total = 45.940 ms GcsInMemoryStore.Get - 372 total (0 active), Execution time: mean = 22.019 us, total = 8.191 ms, Queueing time: mean = 13.765 us, max = 137.899 us, min = 2.227 us, total = 5.121 ms InternalKVGcsService.grpc_server.InternalKVGet - 371 total (0 active), Execution time: mean = 741.885 us, total = 275.239 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 371 total (0 active), Execution time: mean = 54.310 us, total = 20.149 ms, Queueing time: mean = 279.379 us, max = 22.982 ms, min = 9.997 us, total = 103.649 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 312 total (0 active), Execution time: mean = 624.991 us, total = 194.997 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 312 total (0 active), Execution time: mean = 88.959 us, total = 27.755 ms, Queueing time: mean = 106.120 us, max = 225.322 us, min = 22.663 us, total = 33.109 ms GCSServer.deadline_timer.debug_state_dump - 156 total (1 active), Execution time: mean = 1.292 ms, total = 201.518 ms, Queueing time: mean = 65.041 us, max = 163.420 us, min = 14.502 us, total = 10.146 ms GCSServer.deadline_timer.debug_state_event_stats_print - 26 total (1 active, 1 running), Execution time: mean = 1.492 ms, total = 38.786 ms, Queueing time: mean = 59.180 us, max = 137.733 us, min = 14.993 us, total = 1.539 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:38:15,230 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 71990 total (1 active) Queueing time: mean = 46.088 us, max = 3.642 ms, min = 2.014 us, total = 3.318 s Execution time: mean = 237.898 us, total = 17.126 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 32718 total (0 active), Execution time: mean = 493.392 us, total = 16.143 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 32718 total (0 active), Execution time: mean = 25.301 us, total = 827.784 ms, Queueing time: mean = 98.868 us, max = 1.144 ms, min = 4.024 us, total = 3.235 s event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 24.431 us, total = 152.447 ms, Queueing time: mean = 8.696 us, max = 395.058 us, min = 2.014 us, total = 54.264 ms GcsTaskManager.GcJobSummary - 313 total (1 active), Execution time: mean = 10.373 us, total = 3.247 ms, Queueing time: mean = 92.037 us, max = 3.642 ms, min = 16.403 us, total = 28.807 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:38:15,230 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7779 total (25 active) Queueing time: mean = 17.570 us, max = 600.706 us, min = 2.413 us, total = 136.675 ms Execution time: mean = 3.781 s, total = 29410.688 s Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 27.214 us, total = 169.815 ms, Queueing time: mean = 9.929 us, max = 600.706 us, min = 2.413 us, total = 61.958 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 406 total (0 active), Execution time: mean = 39.752 us, total = 16.139 ms, Queueing time: mean = 93.197 us, max = 233.972 us, min = 8.714 us, total = 37.838 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 406 total (24 active), Execution time: mean = 72.439 s, total = 29410.225 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 335 total (0 active), Execution time: mean = 105.051 us, total = 35.192 ms, Queueing time: mean = 102.834 us, max = 205.832 us, min = 11.318 us, total = 34.449 ms InternalPubSubGcsService.grpc_server.GcsPublish - 335 total (0 active), Execution time: mean = 603.960 us, total = 202.326 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 572.887 us, total = 3.437 ms, Queueing time: mean = 72.940 us, max = 114.456 us, min = 54.388 us, total = 437.639 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:38:15,230 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6286 total (0 active) Queueing time: mean = 11.825 us, max = 2.505 ms, min = 563.000 ns, total = 74.334 ms Execution time: mean = 29.212 us, total = 183.625 ms Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 28.837 us, total = 179.940 ms, Queueing time: mean = 11.503 us, max = 2.505 ms, min = 3.133 us, total = 71.778 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:39:15,230 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 324 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 346 - cumulative published bytes: 275155 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:39:15,231 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 19805 total (4 active) Queueing time: mean = 147.729 us, max = 225.583 ms, min = 727.000 ns, total = 2.926 s Execution time: mean = 249.469 us, total = 4.941 s Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 25.108 us, total = 162.703 ms, Queueing time: mean = 10.610 us, max = 6.895 ms, min = 1.268 us, total = 68.750 ms GcsInMemoryStore.Put - 2637 total (0 active), Execution time: mean = 91.038 us, total = 240.066 ms, Queueing time: mean = 640.634 us, max = 225.278 ms, min = 1.495 us, total = 1.689 s RayletLoadPulled - 1620 total (1 active), Execution time: mean = 246.984 us, total = 400.115 ms, Queueing time: mean = 74.901 us, max = 1.351 ms, min = 10.621 us, total = 121.339 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1618 total (0 active), Execution time: mean = 75.630 us, total = 122.369 ms, Queueing time: mean = 98.295 us, max = 5.512 ms, min = 11.600 us, total = 159.042 ms NodeManagerService.grpc_client.GetResourceLoad - 1618 total (0 active), Execution time: mean = 1.440 ms, total = 2.330 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 661 total (0 active), Execution time: mean = 523.316 us, total = 345.912 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 661 total (0 active), Execution time: mean = 40.325 us, total = 26.655 ms, Queueing time: mean = 93.782 us, max = 601.063 us, min = 12.201 us, total = 61.990 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 648 total (0 active), Execution time: mean = 35.090 us, total = 22.738 ms, Queueing time: mean = 105.696 us, max = 283.661 us, min = 17.215 us, total = 68.491 ms NodeInfoGcsService.grpc_server.CheckAlive - 648 total (0 active), Execution time: mean = 535.366 us, total = 346.917 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 541 total (1 active), Execution time: mean = 10.731 us, total = 5.805 ms, Queueing time: mean = 73.465 us, max = 715.636 us, min = 19.990 us, total = 39.745 ms HealthCheck - 538 total (0 active), Execution time: mean = 7.864 us, total = 4.231 ms, Queueing time: mean = 88.653 us, max = 238.751 us, min = 11.335 us, total = 47.695 ms GcsInMemoryStore.Get - 383 total (0 active), Execution time: mean = 21.932 us, total = 8.400 ms, Queueing time: mean = 13.704 us, max = 137.899 us, min = 2.227 us, total = 5.249 ms InternalKVGcsService.grpc_server.InternalKVGet - 382 total (0 active), Execution time: mean = 737.907 us, total = 281.880 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 382 total (0 active), Execution time: mean = 54.462 us, total = 20.804 ms, Queueing time: mean = 273.918 us, max = 22.982 ms, min = 9.997 us, total = 104.637 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 324 total (0 active), Execution time: mean = 626.907 us, total = 203.118 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 324 total (0 active), Execution time: mean = 89.657 us, total = 29.049 ms, Queueing time: mean = 106.405 us, max = 225.322 us, min = 22.663 us, total = 34.475 ms GCSServer.deadline_timer.debug_state_dump - 162 total (1 active), Execution time: mean = 1.292 ms, total = 209.246 ms, Queueing time: mean = 65.550 us, max = 163.420 us, min = 14.502 us, total = 10.619 ms GCSServer.deadline_timer.debug_state_event_stats_print - 27 total (1 active, 1 running), Execution time: mean = 1.502 ms, total = 40.559 ms, Queueing time: mean = 60.389 us, max = 137.733 us, min = 14.993 us, total = 1.631 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:39:15,232 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 74762 total (1 active) Queueing time: mean = 46.064 us, max = 3.642 ms, min = 2.014 us, total = 3.444 s Execution time: mean = 237.886 us, total = 17.785 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 33978 total (0 active), Execution time: mean = 493.345 us, total = 16.763 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 33978 total (0 active), Execution time: mean = 25.318 us, total = 860.247 ms, Queueing time: mean = 98.819 us, max = 1.144 ms, min = 4.024 us, total = 3.358 s event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 24.430 us, total = 158.305 ms, Queueing time: mean = 8.697 us, max = 395.058 us, min = 2.014 us, total = 56.355 ms GcsTaskManager.GcJobSummary - 325 total (1 active), Execution time: mean = 10.446 us, total = 3.395 ms, Queueing time: mean = 91.627 us, max = 3.642 ms, min = 16.403 us, total = 29.779 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:39:15,232 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8063 total (25 active) Queueing time: mean = 17.538 us, max = 600.706 us, min = 2.413 us, total = 141.408 ms Execution time: mean = 3.654 s, total = 29465.825 s Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 27.314 us, total = 176.992 ms, Queueing time: mean = 9.977 us, max = 600.706 us, min = 2.413 us, total = 64.650 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 417 total (0 active), Execution time: mean = 39.717 us, total = 16.562 ms, Queueing time: mean = 92.936 us, max = 233.972 us, min = 8.714 us, total = 38.754 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 417 total (24 active), Execution time: mean = 70.660 s, total = 29465.347 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 346 total (0 active), Execution time: mean = 104.808 us, total = 36.263 ms, Queueing time: mean = 102.818 us, max = 205.832 us, min = 11.318 us, total = 35.575 ms InternalPubSubGcsService.grpc_server.GcsPublish - 346 total (0 active), Execution time: mean = 604.046 us, total = 209.000 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 572.887 us, total = 3.437 ms, Queueing time: mean = 72.940 us, max = 114.456 us, min = 54.388 us, total = 437.639 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:39:15,232 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6526 total (0 active) Queueing time: mean = 11.830 us, max = 2.505 ms, min = 563.000 ns, total = 77.200 ms Execution time: mean = 29.259 us, total = 190.941 ms Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 28.898 us, total = 187.257 ms, Queueing time: mean = 11.519 us, max = 2.505 ms, min = 3.133 us, total = 74.644 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:40:15,232 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 335 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 358 - cumulative published bytes: 284983 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:40:15,233 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 20522 total (4 active) Queueing time: mean = 144.006 us, max = 225.583 ms, min = 727.000 ns, total = 2.955 s Execution time: mean = 249.371 us, total = 5.118 s Event stats: event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 25.181 us, total = 169.216 ms, Queueing time: mean = 10.588 us, max = 6.895 ms, min = 1.268 us, total = 71.149 ms GcsInMemoryStore.Put - 2733 total (0 active), Execution time: mean = 88.034 us, total = 240.597 ms, Queueing time: mean = 619.724 us, max = 225.278 ms, min = 1.495 us, total = 1.694 s RayletLoadPulled - 1680 total (1 active), Execution time: mean = 247.310 us, total = 415.481 ms, Queueing time: mean = 74.712 us, max = 1.351 ms, min = 10.621 us, total = 125.516 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1678 total (0 active), Execution time: mean = 75.639 us, total = 126.922 ms, Queueing time: mean = 98.551 us, max = 5.512 ms, min = 11.600 us, total = 165.368 ms NodeManagerService.grpc_client.GetResourceLoad - 1678 total (0 active), Execution time: mean = 1.444 ms, total = 2.423 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 685 total (0 active), Execution time: mean = 524.244 us, total = 359.107 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 685 total (0 active), Execution time: mean = 40.218 us, total = 27.550 ms, Queueing time: mean = 93.716 us, max = 601.063 us, min = 12.201 us, total = 64.195 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 672 total (0 active), Execution time: mean = 35.126 us, total = 23.604 ms, Queueing time: mean = 106.466 us, max = 283.661 us, min = 17.215 us, total = 71.545 ms NodeInfoGcsService.grpc_server.CheckAlive - 672 total (0 active), Execution time: mean = 538.074 us, total = 361.586 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 561 total (1 active), Execution time: mean = 10.682 us, total = 5.992 ms, Queueing time: mean = 73.664 us, max = 715.636 us, min = 19.990 us, total = 41.326 ms HealthCheck - 558 total (0 active), Execution time: mean = 7.900 us, total = 4.408 ms, Queueing time: mean = 88.250 us, max = 238.751 us, min = 11.335 us, total = 49.244 ms GcsInMemoryStore.Get - 395 total (0 active), Execution time: mean = 22.329 us, total = 8.820 ms, Queueing time: mean = 13.694 us, max = 137.899 us, min = 2.227 us, total = 5.409 ms InternalKVGcsService.grpc_server.InternalKVGet - 394 total (0 active), Execution time: mean = 735.955 us, total = 289.966 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 394 total (0 active), Execution time: mean = 54.788 us, total = 21.586 ms, Queueing time: mean = 270.281 us, max = 22.982 ms, min = 9.997 us, total = 106.491 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 335 total (0 active), Execution time: mean = 628.373 us, total = 210.505 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 335 total (0 active), Execution time: mean = 90.015 us, total = 30.155 ms, Queueing time: mean = 106.473 us, max = 225.322 us, min = 22.663 us, total = 35.668 ms GCSServer.deadline_timer.debug_state_dump - 168 total (1 active), Execution time: mean = 1.287 ms, total = 216.226 ms, Queueing time: mean = 66.782 us, max = 163.420 us, min = 14.502 us, total = 11.219 ms GCSServer.deadline_timer.debug_state_event_stats_print - 28 total (1 active, 1 running), Execution time: mean = 1.512 ms, total = 42.325 ms, Queueing time: mean = 60.464 us, max = 137.733 us, min = 14.993 us, total = 1.693 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:40:15,233 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 77534 total (1 active) Queueing time: mean = 46.236 us, max = 3.642 ms, min = 2.014 us, total = 3.585 s Execution time: mean = 238.573 us, total = 18.498 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 35238 total (0 active), Execution time: mean = 494.772 us, total = 17.435 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 35238 total (0 active), Execution time: mean = 25.397 us, total = 894.928 ms, Queueing time: mean = 99.205 us, max = 1.144 ms, min = 4.024 us, total = 3.496 s event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 24.445 us, total = 164.273 ms, Queueing time: mean = 8.700 us, max = 395.058 us, min = 2.014 us, total = 58.462 ms GcsTaskManager.GcJobSummary - 337 total (1 active), Execution time: mean = 10.411 us, total = 3.508 ms, Queueing time: mean = 90.887 us, max = 3.642 ms, min = 16.403 us, total = 30.629 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:40:15,233 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8351 total (25 active) Queueing time: mean = 17.543 us, max = 600.706 us, min = 2.413 us, total = 146.500 ms Execution time: mean = 3.536 s, total = 29525.959 s Event stats: event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 27.383 us, total = 184.016 ms, Queueing time: mean = 10.003 us, max = 600.706 us, min = 2.413 us, total = 67.219 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 429 total (0 active), Execution time: mean = 39.835 us, total = 17.089 ms, Queueing time: mean = 93.042 us, max = 233.972 us, min = 8.714 us, total = 39.915 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 429 total (24 active), Execution time: mean = 68.824 s, total = 29525.464 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 358 total (0 active), Execution time: mean = 104.949 us, total = 37.572 ms, Queueing time: mean = 103.176 us, max = 205.832 us, min = 11.318 us, total = 36.937 ms InternalPubSubGcsService.grpc_server.GcsPublish - 358 total (0 active), Execution time: mean = 605.032 us, total = 216.601 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 572.887 us, total = 3.437 ms, Queueing time: mean = 72.940 us, max = 114.456 us, min = 54.388 us, total = 437.639 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:40:15,234 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6766 total (0 active) Queueing time: mean = 11.776 us, max = 2.505 ms, min = 563.000 ns, total = 79.678 ms Execution time: mean = 29.219 us, total = 197.696 ms Event stats: event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 28.871 us, total = 194.012 ms, Queueing time: mean = 11.476 us, max = 2.505 ms, min = 3.133 us, total = 77.122 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:41:15,234 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 347 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 370 - cumulative published bytes: 294811 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:41:15,235 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 21240 total (4 active) Queueing time: mean = 140.342 us, max = 225.583 ms, min = 727.000 ns, total = 2.981 s Execution time: mean = 248.617 us, total = 5.281 s Event stats: event_loop_lag_probe - 6959 total (0 active), Execution time: mean = 25.143 us, total = 174.973 ms, Queueing time: mean = 10.516 us, max = 6.895 ms, min = 1.268 us, total = 73.178 ms GcsInMemoryStore.Put - 2829 total (0 active), Execution time: mean = 85.197 us, total = 241.022 ms, Queueing time: mean = 599.954 us, max = 225.278 ms, min = 1.495 us, total = 1.697 s RayletLoadPulled - 1740 total (1 active), Execution time: mean = 247.047 us, total = 429.862 ms, Queueing time: mean = 74.854 us, max = 1.351 ms, min = 10.621 us, total = 130.246 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1738 total (0 active), Execution time: mean = 75.470 us, total = 131.166 ms, Queueing time: mean = 98.152 us, max = 5.512 ms, min = 11.600 us, total = 170.589 ms NodeManagerService.grpc_client.GetResourceLoad - 1738 total (0 active), Execution time: mean = 1.444 ms, total = 2.510 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 709 total (0 active), Execution time: mean = 522.982 us, total = 370.794 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 709 total (0 active), Execution time: mean = 39.927 us, total = 28.308 ms, Queueing time: mean = 93.067 us, max = 601.063 us, min = 12.201 us, total = 65.985 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 696 total (0 active), Execution time: mean = 34.969 us, total = 24.338 ms, Queueing time: mean = 106.540 us, max = 283.661 us, min = 17.215 us, total = 74.152 ms NodeInfoGcsService.grpc_server.CheckAlive - 696 total (0 active), Execution time: mean = 538.682 us, total = 374.922 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 581 total (1 active), Execution time: mean = 10.657 us, total = 6.192 ms, Queueing time: mean = 73.303 us, max = 715.636 us, min = 14.275 us, total = 42.589 ms HealthCheck - 578 total (0 active), Execution time: mean = 7.864 us, total = 4.545 ms, Queueing time: mean = 87.833 us, max = 238.751 us, min = 11.335 us, total = 50.767 ms GcsInMemoryStore.Get - 407 total (0 active), Execution time: mean = 22.205 us, total = 9.038 ms, Queueing time: mean = 13.668 us, max = 137.899 us, min = 2.227 us, total = 5.563 ms InternalKVGcsService.grpc_server.InternalKVGet - 406 total (0 active), Execution time: mean = 729.391 us, total = 296.133 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 406 total (0 active), Execution time: mean = 54.834 us, total = 22.263 ms, Queueing time: mean = 264.875 us, max = 22.982 ms, min = 9.997 us, total = 107.539 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 347 total (0 active), Execution time: mean = 628.080 us, total = 217.944 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 347 total (0 active), Execution time: mean = 89.626 us, total = 31.100 ms, Queueing time: mean = 106.303 us, max = 225.322 us, min = 17.034 us, total = 36.887 ms GCSServer.deadline_timer.debug_state_dump - 174 total (1 active), Execution time: mean = 1.282 ms, total = 223.024 ms, Queueing time: mean = 66.582 us, max = 163.420 us, min = 14.502 us, total = 11.585 ms GCSServer.deadline_timer.debug_state_event_stats_print - 29 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 44.041 ms, Queueing time: mean = 60.481 us, max = 137.733 us, min = 14.993 us, total = 1.754 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:41:15,235 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 80305 total (1 active) Queueing time: mean = 46.223 us, max = 3.642 ms, min = 2.014 us, total = 3.712 s Execution time: mean = 238.300 us, total = 19.137 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 36498 total (0 active), Execution time: mean = 494.231 us, total = 18.038 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 36498 total (0 active), Execution time: mean = 25.335 us, total = 924.687 ms, Queueing time: mean = 99.185 us, max = 1.144 ms, min = 4.024 us, total = 3.620 s event_loop_lag_probe - 6959 total (0 active), Execution time: mean = 24.417 us, total = 169.919 ms, Queueing time: mean = 8.679 us, max = 395.058 us, min = 2.014 us, total = 60.397 ms GcsTaskManager.GcJobSummary - 349 total (1 active), Execution time: mean = 10.340 us, total = 3.609 ms, Queueing time: mean = 90.094 us, max = 3.642 ms, min = 14.860 us, total = 31.443 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:41:15,235 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8639 total (25 active) Queueing time: mean = 17.474 us, max = 600.706 us, min = 2.413 us, total = 150.955 ms Execution time: mean = 3.425 s, total = 29586.182 s Event stats: event_loop_lag_probe - 6960 total (0 active), Execution time: mean = 27.304 us, total = 190.037 ms, Queueing time: mean = 9.978 us, max = 600.706 us, min = 2.413 us, total = 69.447 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 441 total (0 active), Execution time: mean = 39.888 us, total = 17.591 ms, Queueing time: mean = 93.235 us, max = 233.972 us, min = 8.714 us, total = 41.117 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 441 total (24 active), Execution time: mean = 67.088 s, total = 29585.674 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 370 total (0 active), Execution time: mean = 104.201 us, total = 38.554 ms, Queueing time: mean = 102.601 us, max = 205.832 us, min = 11.318 us, total = 37.962 ms InternalPubSubGcsService.grpc_server.GcsPublish - 370 total (0 active), Execution time: mean = 602.284 us, total = 222.845 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 572.887 us, total = 3.437 ms, Queueing time: mean = 72.940 us, max = 114.456 us, min = 54.388 us, total = 437.639 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:41:15,235 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7006 total (0 active) Queueing time: mean = 11.707 us, max = 2.505 ms, min = 563.000 ns, total = 82.016 ms Execution time: mean = 29.230 us, total = 204.786 ms Event stats: event_loop_lag_probe - 6960 total (0 active), Execution time: mean = 28.894 us, total = 201.102 ms, Queueing time: mean = 11.417 us, max = 2.505 ms, min = 3.133 us, total = 79.460 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:42:15,235 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 359 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 382 - cumulative published bytes: 304639 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:42:15,236 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 21959 total (4 active) Queueing time: mean = 136.929 us, max = 225.583 ms, min = -0.000 s, total = 3.007 s Execution time: mean = 247.265 us, total = 5.430 s Event stats: event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 25.071 us, total = 180.487 ms, Queueing time: mean = 10.451 us, max = 6.895 ms, min = 1.268 us, total = 75.236 ms GcsInMemoryStore.Put - 2925 total (0 active), Execution time: mean = 82.555 us, total = 241.474 ms, Queueing time: mean = 581.912 us, max = 225.278 ms, min = 1.495 us, total = 1.702 s RayletLoadPulled - 1800 total (1 active), Execution time: mean = 246.664 us, total = 443.995 ms, Queueing time: mean = 74.695 us, max = 1.351 ms, min = 10.621 us, total = 134.452 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1798 total (0 active), Execution time: mean = 75.484 us, total = 135.721 ms, Queueing time: mean = 97.690 us, max = 5.512 ms, min = 11.600 us, total = 175.647 ms NodeManagerService.grpc_client.GetResourceLoad - 1798 total (0 active), Execution time: mean = 1.439 ms, total = 2.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 733 total (0 active), Execution time: mean = 521.389 us, total = 382.178 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 733 total (0 active), Execution time: mean = 39.826 us, total = 29.192 ms, Queueing time: mean = 92.752 us, max = 601.063 us, min = 12.201 us, total = 67.987 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 720 total (0 active), Execution time: mean = 34.999 us, total = 25.199 ms, Queueing time: mean = 105.936 us, max = 283.661 us, min = 17.215 us, total = 76.274 ms NodeInfoGcsService.grpc_server.CheckAlive - 720 total (0 active), Execution time: mean = 535.589 us, total = 385.624 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 601 total (1 active), Execution time: mean = 10.629 us, total = 6.388 ms, Queueing time: mean = 73.267 us, max = 715.636 us, min = 14.275 us, total = 44.034 ms HealthCheck - 598 total (0 active), Execution time: mean = 7.818 us, total = 4.675 ms, Queueing time: mean = 87.278 us, max = 238.751 us, min = 11.335 us, total = 52.192 ms GcsInMemoryStore.Get - 419 total (0 active), Execution time: mean = 22.037 us, total = 9.234 ms, Queueing time: mean = 13.582 us, max = 137.899 us, min = 2.227 us, total = 5.691 ms InternalKVGcsService.grpc_server.InternalKVGet - 418 total (0 active), Execution time: mean = 721.652 us, total = 301.650 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 418 total (0 active), Execution time: mean = 54.784 us, total = 22.900 ms, Queueing time: mean = 259.646 us, max = 22.982 ms, min = 9.997 us, total = 108.532 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 359 total (0 active), Execution time: mean = 626.809 us, total = 225.025 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 359 total (0 active), Execution time: mean = 90.095 us, total = 32.344 ms, Queueing time: mean = 106.609 us, max = 225.322 us, min = 17.034 us, total = 38.273 ms GCSServer.deadline_timer.debug_state_dump - 180 total (1 active), Execution time: mean = 1.278 ms, total = 229.963 ms, Queueing time: mean = 65.927 us, max = 163.420 us, min = -0.000 s, total = 11.867 ms GCSServer.deadline_timer.debug_state_event_stats_print - 30 total (1 active, 1 running), Execution time: mean = 1.517 ms, total = 45.495 ms, Queueing time: mean = 59.134 us, max = 137.733 us, min = 14.993 us, total = 1.774 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:42:15,236 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 83077 total (1 active) Queueing time: mean = 45.916 us, max = 3.642 ms, min = 2.014 us, total = 3.815 s Execution time: mean = 236.898 us, total = 19.681 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 37758 total (0 active), Execution time: mean = 491.281 us, total = 18.550 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 37758 total (0 active), Execution time: mean = 25.214 us, total = 952.032 ms, Queueing time: mean = 98.524 us, max = 1.144 ms, min = 4.024 us, total = 3.720 s event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 24.344 us, total = 175.253 ms, Queueing time: mean = 8.651 us, max = 395.058 us, min = 2.014 us, total = 62.282 ms GcsTaskManager.GcJobSummary - 361 total (1 active), Execution time: mean = 10.277 us, total = 3.710 ms, Queueing time: mean = 89.163 us, max = 3.642 ms, min = 14.860 us, total = 32.188 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:42:15,237 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8971 total (25 active) Queueing time: mean = 17.369 us, max = 600.706 us, min = 2.413 us, total = 155.814 ms Execution time: mean = 4.776 s, total = 42846.271 s Event stats: event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 27.132 us, total = 195.320 ms, Queueing time: mean = 9.915 us, max = 600.706 us, min = 2.413 us, total = 71.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 475 total (0 active), Execution time: mean = 38.347 us, total = 18.215 ms, Queueing time: mean = 90.351 us, max = 233.972 us, min = 5.417 us, total = 42.917 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 475 total (24 active), Execution time: mean = 90.202 s, total = 42845.749 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 382 total (0 active), Execution time: mean = 103.632 us, total = 39.587 ms, Queueing time: mean = 102.157 us, max = 205.832 us, min = 11.318 us, total = 39.024 ms InternalPubSubGcsService.grpc_server.GcsPublish - 382 total (0 active), Execution time: mean = 598.924 us, total = 228.789 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 581.627 us, total = 4.071 ms, Queueing time: mean = 72.235 us, max = 114.456 us, min = 54.388 us, total = 505.644 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:42:15,237 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7245 total (0 active) Queueing time: mean = 11.634 us, max = 2.505 ms, min = 563.000 ns, total = 84.290 ms Execution time: mean = 29.104 us, total = 210.861 ms Event stats: event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 28.779 us, total = 207.177 ms, Queueing time: mean = 11.354 us, max = 2.505 ms, min = 3.133 us, total = 81.734 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:43:15,237 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 371 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 394 - cumulative published bytes: 314467 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:43:15,238 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 22678 total (4 active) Queueing time: mean = 133.699 us, max = 225.583 ms, min = -0.000 s, total = 3.032 s Execution time: mean = 246.400 us, total = 5.588 s Event stats: event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 25.103 us, total = 186.742 ms, Queueing time: mean = 10.438 us, max = 6.895 ms, min = 1.268 us, total = 77.645 ms GcsInMemoryStore.Put - 3021 total (0 active), Execution time: mean = 80.071 us, total = 241.895 ms, Queueing time: mean = 564.831 us, max = 225.278 ms, min = 1.495 us, total = 1.706 s RayletLoadPulled - 1860 total (1 active), Execution time: mean = 246.474 us, total = 458.442 ms, Queueing time: mean = 74.605 us, max = 1.351 ms, min = 10.621 us, total = 138.766 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1858 total (0 active), Execution time: mean = 75.448 us, total = 140.182 ms, Queueing time: mean = 97.067 us, max = 5.512 ms, min = 11.600 us, total = 180.351 ms NodeManagerService.grpc_client.GetResourceLoad - 1858 total (0 active), Execution time: mean = 1.437 ms, total = 2.669 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 757 total (0 active), Execution time: mean = 520.426 us, total = 393.963 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 757 total (0 active), Execution time: mean = 39.844 us, total = 30.162 ms, Queueing time: mean = 92.293 us, max = 601.063 us, min = 12.201 us, total = 69.866 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 744 total (0 active), Execution time: mean = 34.939 us, total = 25.995 ms, Queueing time: mean = 105.043 us, max = 283.661 us, min = 17.215 us, total = 78.152 ms NodeInfoGcsService.grpc_server.CheckAlive - 744 total (0 active), Execution time: mean = 534.425 us, total = 397.612 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 621 total (1 active), Execution time: mean = 10.622 us, total = 6.596 ms, Queueing time: mean = 73.200 us, max = 715.636 us, min = 14.275 us, total = 45.457 ms HealthCheck - 618 total (0 active), Execution time: mean = 7.824 us, total = 4.835 ms, Queueing time: mean = 87.258 us, max = 238.751 us, min = 11.335 us, total = 53.926 ms GcsInMemoryStore.Get - 431 total (0 active), Execution time: mean = 21.968 us, total = 9.468 ms, Queueing time: mean = 13.527 us, max = 137.899 us, min = 2.227 us, total = 5.830 ms InternalKVGcsService.grpc_server.InternalKVGet - 430 total (0 active), Execution time: mean = 716.501 us, total = 308.096 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 430 total (0 active), Execution time: mean = 54.666 us, total = 23.506 ms, Queueing time: mean = 254.593 us, max = 22.982 ms, min = 9.997 us, total = 109.475 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 371 total (0 active), Execution time: mean = 624.364 us, total = 231.639 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 371 total (0 active), Execution time: mean = 89.966 us, total = 33.377 ms, Queueing time: mean = 105.990 us, max = 225.322 us, min = 17.034 us, total = 39.322 ms GCSServer.deadline_timer.debug_state_dump - 186 total (1 active), Execution time: mean = 1.282 ms, total = 238.421 ms, Queueing time: mean = 66.096 us, max = 163.420 us, min = -0.000 s, total = 12.294 ms GCSServer.deadline_timer.debug_state_event_stats_print - 31 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 47.101 ms, Queueing time: mean = 58.937 us, max = 137.733 us, min = 14.993 us, total = 1.827 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:43:15,238 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 85849 total (1 active) Queueing time: mean = 45.819 us, max = 3.642 ms, min = 2.014 us, total = 3.933 s Execution time: mean = 236.547 us, total = 20.307 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 39018 total (0 active), Execution time: mean = 490.578 us, total = 19.141 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 39018 total (0 active), Execution time: mean = 25.152 us, total = 981.399 ms, Queueing time: mean = 98.321 us, max = 1.144 ms, min = 4.024 us, total = 3.836 s event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 24.296 us, total = 180.738 ms, Queueing time: mean = 8.630 us, max = 395.058 us, min = 2.014 us, total = 64.202 ms GcsTaskManager.GcJobSummary - 373 total (1 active), Execution time: mean = 10.282 us, total = 3.835 ms, Queueing time: mean = 88.449 us, max = 3.642 ms, min = 14.860 us, total = 32.991 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:43:15,238 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9259 total (25 active) Queueing time: mean = 17.296 us, max = 600.706 us, min = 2.413 us, total = 160.147 ms Execution time: mean = 4.634 s, total = 42906.408 s Event stats: event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 27.015 us, total = 200.966 ms, Queueing time: mean = 9.870 us, max = 600.706 us, min = 2.413 us, total = 73.422 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 487 total (0 active), Execution time: mean = 38.523 us, total = 18.761 ms, Queueing time: mean = 90.367 us, max = 233.972 us, min = 5.417 us, total = 44.009 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 487 total (24 active), Execution time: mean = 88.102 s, total = 42905.872 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 394 total (0 active), Execution time: mean = 103.753 us, total = 40.878 ms, Queueing time: mean = 102.079 us, max = 205.832 us, min = 11.318 us, total = 40.219 ms InternalPubSubGcsService.grpc_server.GcsPublish - 394 total (0 active), Execution time: mean = 597.988 us, total = 235.607 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 581.627 us, total = 4.071 ms, Queueing time: mean = 72.235 us, max = 114.456 us, min = 54.388 us, total = 505.644 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:43:15,239 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7485 total (0 active) Queueing time: mean = 11.568 us, max = 2.505 ms, min = 563.000 ns, total = 86.588 ms Execution time: mean = 29.010 us, total = 217.143 ms Event stats: event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 28.695 us, total = 213.459 ms, Queueing time: mean = 11.296 us, max = 2.505 ms, min = 3.043 us, total = 84.032 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:44:15,239 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 383 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 407 - cumulative published bytes: 324329 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:44:15,240 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 23397 total (4 active) Queueing time: mean = 130.750 us, max = 225.583 ms, min = -0.000 s, total = 3.059 s Execution time: mean = 246.075 us, total = 5.757 s Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 25.132 us, total = 192.988 ms, Queueing time: mean = 10.413 us, max = 6.895 ms, min = 1.268 us, total = 79.961 ms GcsInMemoryStore.Put - 3117 total (0 active), Execution time: mean = 77.746 us, total = 242.333 ms, Queueing time: mean = 548.884 us, max = 225.278 ms, min = 1.495 us, total = 1.711 s RayletLoadPulled - 1920 total (1 active), Execution time: mean = 246.369 us, total = 473.029 ms, Queueing time: mean = 74.718 us, max = 1.351 ms, min = 10.621 us, total = 143.458 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1918 total (0 active), Execution time: mean = 75.478 us, total = 144.766 ms, Queueing time: mean = 96.735 us, max = 5.512 ms, min = 11.600 us, total = 185.537 ms NodeManagerService.grpc_client.GetResourceLoad - 1918 total (0 active), Execution time: mean = 1.438 ms, total = 2.758 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 781 total (0 active), Execution time: mean = 520.574 us, total = 406.568 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 781 total (0 active), Execution time: mean = 39.810 us, total = 31.092 ms, Queueing time: mean = 91.740 us, max = 601.063 us, min = 12.201 us, total = 71.649 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 768 total (0 active), Execution time: mean = 34.839 us, total = 26.757 ms, Queueing time: mean = 104.869 us, max = 283.661 us, min = 17.215 us, total = 80.540 ms NodeInfoGcsService.grpc_server.CheckAlive - 768 total (0 active), Execution time: mean = 535.246 us, total = 411.069 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 641 total (1 active), Execution time: mean = 10.618 us, total = 6.806 ms, Queueing time: mean = 73.371 us, max = 715.636 us, min = 14.275 us, total = 47.030 ms HealthCheck - 638 total (0 active), Execution time: mean = 7.801 us, total = 4.977 ms, Queueing time: mean = 87.381 us, max = 238.751 us, min = 11.335 us, total = 55.749 ms GcsInMemoryStore.Get - 443 total (0 active), Execution time: mean = 21.841 us, total = 9.675 ms, Queueing time: mean = 13.469 us, max = 137.899 us, min = 2.227 us, total = 5.967 ms InternalKVGcsService.grpc_server.InternalKVGet - 442 total (0 active), Execution time: mean = 712.598 us, total = 314.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 442 total (0 active), Execution time: mean = 54.803 us, total = 24.223 ms, Queueing time: mean = 249.849 us, max = 22.982 ms, min = 9.997 us, total = 110.433 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 383 total (0 active), Execution time: mean = 624.736 us, total = 239.274 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 383 total (0 active), Execution time: mean = 89.761 us, total = 34.378 ms, Queueing time: mean = 105.604 us, max = 225.322 us, min = 17.034 us, total = 40.446 ms GCSServer.deadline_timer.debug_state_dump - 192 total (1 active), Execution time: mean = 1.288 ms, total = 247.284 ms, Queueing time: mean = 66.889 us, max = 163.420 us, min = -0.000 s, total = 12.843 ms GCSServer.deadline_timer.debug_state_event_stats_print - 32 total (1 active, 1 running), Execution time: mean = 1.526 ms, total = 48.821 ms, Queueing time: mean = 59.670 us, max = 137.733 us, min = 14.993 us, total = 1.909 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:44:15,240 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 88621 total (1 active) Queueing time: mean = 45.964 us, max = 3.642 ms, min = 2.014 us, total = 4.073 s Execution time: mean = 237.044 us, total = 21.007 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 40278 total (0 active), Execution time: mean = 491.703 us, total = 19.805 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 40278 total (0 active), Execution time: mean = 25.118 us, total = 1.012 s, Queueing time: mean = 98.647 us, max = 1.144 ms, min = 4.024 us, total = 3.973 s event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 24.290 us, total = 186.524 ms, Queueing time: mean = 8.627 us, max = 395.058 us, min = 2.014 us, total = 66.246 ms GcsTaskManager.GcJobSummary - 385 total (1 active), Execution time: mean = 10.312 us, total = 3.970 ms, Queueing time: mean = 87.878 us, max = 3.642 ms, min = 14.860 us, total = 33.833 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:44:15,240 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9551 total (25 active) Queueing time: mean = 17.310 us, max = 600.706 us, min = 2.413 us, total = 165.331 ms Execution time: mean = 4.499 s, total = 42966.653 s Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 26.997 us, total = 207.307 ms, Queueing time: mean = 9.860 us, max = 600.706 us, min = 2.413 us, total = 75.714 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 500 total (0 active), Execution time: mean = 38.624 us, total = 19.312 ms, Queueing time: mean = 91.308 us, max = 233.972 us, min = 5.417 us, total = 45.654 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 500 total (24 active), Execution time: mean = 85.932 s, total = 42966.100 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 407 total (0 active), Execution time: mean = 104.138 us, total = 42.384 ms, Queueing time: mean = 101.884 us, max = 205.832 us, min = 11.318 us, total = 41.467 ms InternalPubSubGcsService.grpc_server.GcsPublish - 407 total (0 active), Execution time: mean = 598.752 us, total = 243.692 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 581.627 us, total = 4.071 ms, Queueing time: mean = 72.235 us, max = 114.456 us, min = 54.388 us, total = 505.644 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:44:15,240 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7725 total (0 active) Queueing time: mean = 11.519 us, max = 2.505 ms, min = 563.000 ns, total = 88.982 ms Execution time: mean = 28.958 us, total = 223.701 ms Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 28.652 us, total = 220.016 ms, Queueing time: mean = 11.255 us, max = 2.505 ms, min = 3.043 us, total = 86.426 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:45:15,240 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 395 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 420 - cumulative published bytes: 334191 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:45:15,241 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 24116 total (4 active) Queueing time: mean = 127.987 us, max = 225.583 ms, min = -0.000 s, total = 3.087 s Execution time: mean = 245.619 us, total = 5.923 s Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 25.080 us, total = 198.607 ms, Queueing time: mean = 10.371 us, max = 6.895 ms, min = 1.268 us, total = 82.128 ms GcsInMemoryStore.Put - 3213 total (0 active), Execution time: mean = 75.569 us, total = 242.804 ms, Queueing time: mean = 534.025 us, max = 225.278 ms, min = 1.495 us, total = 1.716 s RayletLoadPulled - 1980 total (1 active), Execution time: mean = 246.002 us, total = 487.085 ms, Queueing time: mean = 74.709 us, max = 1.351 ms, min = 10.621 us, total = 147.923 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1978 total (0 active), Execution time: mean = 75.401 us, total = 149.143 ms, Queueing time: mean = 96.838 us, max = 5.512 ms, min = 11.600 us, total = 191.545 ms NodeManagerService.grpc_client.GetResourceLoad - 1978 total (0 active), Execution time: mean = 1.438 ms, total = 2.845 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 805 total (0 active), Execution time: mean = 520.624 us, total = 419.102 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 805 total (0 active), Execution time: mean = 39.564 us, total = 31.849 ms, Queueing time: mean = 91.418 us, max = 601.063 us, min = 12.201 us, total = 73.592 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 792 total (0 active), Execution time: mean = 34.967 us, total = 27.694 ms, Queueing time: mean = 104.649 us, max = 283.661 us, min = 17.215 us, total = 82.882 ms NodeInfoGcsService.grpc_server.CheckAlive - 792 total (0 active), Execution time: mean = 535.505 us, total = 424.120 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 661 total (1 active), Execution time: mean = 10.711 us, total = 7.080 ms, Queueing time: mean = 73.167 us, max = 715.636 us, min = 14.275 us, total = 48.363 ms HealthCheck - 658 total (0 active), Execution time: mean = 7.766 us, total = 5.110 ms, Queueing time: mean = 87.143 us, max = 238.751 us, min = 11.335 us, total = 57.340 ms GcsInMemoryStore.Get - 455 total (0 active), Execution time: mean = 21.763 us, total = 9.902 ms, Queueing time: mean = 13.394 us, max = 137.899 us, min = 2.227 us, total = 6.094 ms InternalKVGcsService.grpc_server.InternalKVGet - 454 total (0 active), Execution time: mean = 708.623 us, total = 321.715 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 454 total (0 active), Execution time: mean = 54.655 us, total = 24.813 ms, Queueing time: mean = 245.363 us, max = 22.982 ms, min = 9.997 us, total = 111.395 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 395 total (0 active), Execution time: mean = 624.284 us, total = 246.592 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 395 total (0 active), Execution time: mean = 89.880 us, total = 35.502 ms, Queueing time: mean = 105.120 us, max = 225.322 us, min = 17.034 us, total = 41.523 ms GCSServer.deadline_timer.debug_state_dump - 198 total (1 active), Execution time: mean = 1.293 ms, total = 256.098 ms, Queueing time: mean = 66.860 us, max = 163.420 us, min = -0.000 s, total = 13.238 ms GCSServer.deadline_timer.debug_state_event_stats_print - 33 total (1 active, 1 running), Execution time: mean = 1.529 ms, total = 50.461 ms, Queueing time: mean = 58.522 us, max = 137.733 us, min = 14.993 us, total = 1.931 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:45:15,242 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 91393 total (1 active) Queueing time: mean = 46.100 us, max = 3.642 ms, min = 2.014 us, total = 4.213 s Execution time: mean = 237.372 us, total = 21.694 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 41538 total (0 active), Execution time: mean = 492.421 us, total = 20.454 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 41538 total (0 active), Execution time: mean = 25.128 us, total = 1.044 s, Queueing time: mean = 98.950 us, max = 1.144 ms, min = 4.024 us, total = 4.110 s event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 24.250 us, total = 192.037 ms, Queueing time: mean = 8.617 us, max = 395.058 us, min = 2.014 us, total = 68.234 ms GcsTaskManager.GcJobSummary - 397 total (1 active), Execution time: mean = 10.305 us, total = 4.091 ms, Queueing time: mean = 87.692 us, max = 3.642 ms, min = 14.860 us, total = 34.814 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:45:15,242 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9843 total (25 active) Queueing time: mean = 17.363 us, max = 600.706 us, min = 2.413 us, total = 170.901 ms Execution time: mean = 4.371 s, total = 43026.810 s Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 27.014 us, total = 213.925 ms, Queueing time: mean = 9.869 us, max = 600.706 us, min = 2.413 us, total = 78.154 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 513 total (0 active), Execution time: mean = 38.602 us, total = 19.803 ms, Queueing time: mean = 91.790 us, max = 233.972 us, min = 5.417 us, total = 47.088 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 513 total (24 active), Execution time: mean = 83.872 s, total = 43026.241 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 420 total (0 active), Execution time: mean = 104.257 us, total = 43.788 ms, Queueing time: mean = 102.765 us, max = 205.832 us, min = 11.318 us, total = 43.161 ms InternalPubSubGcsService.grpc_server.GcsPublish - 420 total (0 active), Execution time: mean = 600.173 us, total = 252.073 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 581.627 us, total = 4.071 ms, Queueing time: mean = 72.235 us, max = 114.456 us, min = 54.388 us, total = 505.644 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:45:15,242 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7965 total (0 active) Queueing time: mean = 11.450 us, max = 2.505 ms, min = 563.000 ns, total = 91.199 ms Execution time: mean = 28.835 us, total = 229.671 ms Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 28.537 us, total = 225.987 ms, Queueing time: mean = 11.194 us, max = 2.505 ms, min = 3.043 us, total = 88.643 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:46:15,242 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 407 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 432 - cumulative published bytes: 344019 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:46:15,243 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 24835 total (4 active) Queueing time: mean = 125.382 us, max = 225.583 ms, min = -0.000 s, total = 3.114 s Execution time: mean = 244.643 us, total = 6.076 s Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 25.072 us, total = 204.562 ms, Queueing time: mean = 10.341 us, max = 6.895 ms, min = 1.268 us, total = 84.372 ms GcsInMemoryStore.Put - 3309 total (0 active), Execution time: mean = 73.511 us, total = 243.249 ms, Queueing time: mean = 519.820 us, max = 225.278 ms, min = 1.495 us, total = 1.720 s RayletLoadPulled - 2040 total (1 active), Execution time: mean = 245.551 us, total = 500.924 ms, Queueing time: mean = 74.557 us, max = 1.351 ms, min = 10.621 us, total = 152.097 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2038 total (0 active), Execution time: mean = 75.233 us, total = 153.325 ms, Queueing time: mean = 96.539 us, max = 5.512 ms, min = 11.600 us, total = 196.747 ms NodeManagerService.grpc_client.GetResourceLoad - 2038 total (0 active), Execution time: mean = 1.434 ms, total = 2.923 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 829 total (0 active), Execution time: mean = 518.868 us, total = 430.142 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 829 total (0 active), Execution time: mean = 39.503 us, total = 32.748 ms, Queueing time: mean = 91.329 us, max = 601.063 us, min = 12.201 us, total = 75.712 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 816 total (0 active), Execution time: mean = 34.973 us, total = 28.538 ms, Queueing time: mean = 104.705 us, max = 283.661 us, min = 17.215 us, total = 85.439 ms NodeInfoGcsService.grpc_server.CheckAlive - 816 total (0 active), Execution time: mean = 535.167 us, total = 436.697 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 681 total (1 active), Execution time: mean = 10.677 us, total = 7.271 ms, Queueing time: mean = 73.279 us, max = 715.636 us, min = 14.275 us, total = 49.903 ms HealthCheck - 678 total (0 active), Execution time: mean = 7.738 us, total = 5.246 ms, Queueing time: mean = 87.570 us, max = 238.751 us, min = 11.335 us, total = 59.372 ms GcsInMemoryStore.Get - 467 total (0 active), Execution time: mean = 21.678 us, total = 10.124 ms, Queueing time: mean = 13.503 us, max = 137.899 us, min = 2.227 us, total = 6.306 ms InternalKVGcsService.grpc_server.InternalKVGet - 466 total (0 active), Execution time: mean = 702.684 us, total = 327.451 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 466 total (0 active), Execution time: mean = 54.606 us, total = 25.446 ms, Queueing time: mean = 241.424 us, max = 22.982 ms, min = 9.997 us, total = 112.504 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 407 total (0 active), Execution time: mean = 623.681 us, total = 253.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 407 total (0 active), Execution time: mean = 89.679 us, total = 36.499 ms, Queueing time: mean = 105.315 us, max = 225.322 us, min = 17.034 us, total = 42.863 ms GCSServer.deadline_timer.debug_state_dump - 204 total (1 active), Execution time: mean = 1.293 ms, total = 263.809 ms, Queueing time: mean = 67.107 us, max = 163.420 us, min = -0.000 s, total = 13.690 ms GCSServer.deadline_timer.debug_state_event_stats_print - 34 total (1 active, 1 running), Execution time: mean = 1.534 ms, total = 52.159 ms, Queueing time: mean = 58.756 us, max = 137.733 us, min = 14.993 us, total = 1.998 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:46:15,243 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 94165 total (1 active) Queueing time: mean = 45.812 us, max = 3.642 ms, min = 2.014 us, total = 4.314 s Execution time: mean = 236.374 us, total = 22.258 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 42798 total (0 active), Execution time: mean = 490.328 us, total = 20.985 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 42798 total (0 active), Execution time: mean = 25.029 us, total = 1.071 s, Queueing time: mean = 98.325 us, max = 1.144 ms, min = 4.024 us, total = 4.208 s event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 24.223 us, total = 197.634 ms, Queueing time: mean = 8.604 us, max = 395.058 us, min = 2.014 us, total = 70.197 ms GcsTaskManager.GcJobSummary - 409 total (1 active), Execution time: mean = 10.312 us, total = 4.218 ms, Queueing time: mean = 86.953 us, max = 3.642 ms, min = 14.860 us, total = 35.564 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:46:15,244 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10131 total (25 active) Queueing time: mean = 17.286 us, max = 600.706 us, min = 2.413 us, total = 175.123 ms Execution time: mean = 4.253 s, total = 43086.967 s Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 27.034 us, total = 220.569 ms, Queueing time: mean = 9.871 us, max = 600.706 us, min = 2.413 us, total = 80.535 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 525 total (0 active), Execution time: mean = 38.544 us, total = 20.236 ms, Queueing time: mean = 91.108 us, max = 233.972 us, min = 5.417 us, total = 47.832 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 525 total (24 active), Execution time: mean = 82.069 s, total = 43086.382 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 432 total (0 active), Execution time: mean = 104.524 us, total = 45.154 ms, Queueing time: mean = 102.451 us, max = 205.832 us, min = 11.318 us, total = 44.259 ms InternalPubSubGcsService.grpc_server.GcsPublish - 432 total (0 active), Execution time: mean = 598.069 us, total = 258.366 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 581.627 us, total = 4.071 ms, Queueing time: mean = 72.235 us, max = 114.456 us, min = 54.388 us, total = 505.644 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:46:15,244 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8205 total (0 active) Queueing time: mean = 11.389 us, max = 2.505 ms, min = 563.000 ns, total = 93.444 ms Execution time: mean = 28.727 us, total = 235.702 ms Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 28.437 us, total = 232.018 ms, Queueing time: mean = 11.140 us, max = 2.505 ms, min = 3.043 us, total = 90.888 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:47:15,244 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 419 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 444 - cumulative published bytes: 353847 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:47:15,245 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 25554 total (4 active) Queueing time: mean = 122.651 us, max = 225.583 ms, min = -0.000 s, total = 3.134 s Execution time: mean = 243.880 us, total = 6.232 s Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 25.022 us, total = 210.164 ms, Queueing time: mean = 10.293 us, max = 6.895 ms, min = 1.268 us, total = 86.453 ms GcsInMemoryStore.Put - 3405 total (0 active), Execution time: mean = 71.537 us, total = 243.583 ms, Queueing time: mean = 506.208 us, max = 225.278 ms, min = 1.495 us, total = 1.724 s RayletLoadPulled - 2100 total (1 active), Execution time: mean = 245.368 us, total = 515.273 ms, Queueing time: mean = 74.303 us, max = 1.351 ms, min = 10.621 us, total = 156.036 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2098 total (0 active), Execution time: mean = 75.027 us, total = 157.406 ms, Queueing time: mean = 95.416 us, max = 5.512 ms, min = 11.600 us, total = 200.182 ms NodeManagerService.grpc_client.GetResourceLoad - 2098 total (0 active), Execution time: mean = 1.433 ms, total = 3.007 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 853 total (0 active), Execution time: mean = 517.162 us, total = 441.139 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 853 total (0 active), Execution time: mean = 39.136 us, total = 33.383 ms, Queueing time: mean = 90.206 us, max = 601.063 us, min = 12.201 us, total = 76.946 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 840 total (0 active), Execution time: mean = 34.759 us, total = 29.198 ms, Queueing time: mean = 103.741 us, max = 283.661 us, min = 17.215 us, total = 87.143 ms NodeInfoGcsService.grpc_server.CheckAlive - 840 total (0 active), Execution time: mean = 533.436 us, total = 448.086 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 701 total (1 active), Execution time: mean = 10.648 us, total = 7.464 ms, Queueing time: mean = 72.858 us, max = 715.636 us, min = 14.275 us, total = 51.074 ms HealthCheck - 698 total (0 active), Execution time: mean = 7.686 us, total = 5.365 ms, Queueing time: mean = 86.871 us, max = 238.751 us, min = 11.335 us, total = 60.636 ms GcsInMemoryStore.Get - 479 total (0 active), Execution time: mean = 21.826 us, total = 10.455 ms, Queueing time: mean = 13.472 us, max = 137.899 us, min = 2.227 us, total = 6.453 ms InternalKVGcsService.grpc_server.InternalKVGet - 478 total (0 active), Execution time: mean = 698.022 us, total = 333.654 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 478 total (0 active), Execution time: mean = 54.304 us, total = 25.957 ms, Queueing time: mean = 236.686 us, max = 22.982 ms, min = 9.997 us, total = 113.136 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 419 total (0 active), Execution time: mean = 622.135 us, total = 260.675 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 419 total (0 active), Execution time: mean = 89.290 us, total = 37.412 ms, Queueing time: mean = 104.506 us, max = 225.322 us, min = 17.034 us, total = 43.788 ms GCSServer.deadline_timer.debug_state_dump - 210 total (1 active), Execution time: mean = 1.290 ms, total = 270.909 ms, Queueing time: mean = 66.410 us, max = 163.420 us, min = -0.000 s, total = 13.946 ms GCSServer.deadline_timer.debug_state_event_stats_print - 35 total (1 active, 1 running), Execution time: mean = 1.542 ms, total = 53.961 ms, Queueing time: mean = 57.908 us, max = 137.733 us, min = 14.993 us, total = 2.027 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:47:15,245 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 96937 total (1 active) Queueing time: mean = 45.872 us, max = 3.642 ms, min = 2.014 us, total = 4.447 s Execution time: mean = 236.372 us, total = 22.913 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 44058 total (0 active), Execution time: mean = 490.384 us, total = 21.605 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 44058 total (0 active), Execution time: mean = 24.982 us, total = 1.101 s, Queueing time: mean = 98.469 us, max = 1.144 ms, min = 4.024 us, total = 4.338 s event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 24.156 us, total = 202.882 ms, Queueing time: mean = 8.582 us, max = 395.058 us, min = 2.014 us, total = 72.081 ms GcsTaskManager.GcJobSummary - 421 total (1 active), Execution time: mean = 10.276 us, total = 4.326 ms, Queueing time: mean = 86.241 us, max = 3.642 ms, min = 14.860 us, total = 36.307 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:47:15,245 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10422 total (25 active) Queueing time: mean = 17.231 us, max = 600.706 us, min = 2.413 us, total = 179.577 ms Execution time: mean = 4.198 s, total = 43747.084 s Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 26.930 us, total = 226.184 ms, Queueing time: mean = 9.829 us, max = 600.706 us, min = 2.413 us, total = 82.556 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 538 total (0 active), Execution time: mean = 38.571 us, total = 20.751 ms, Queueing time: mean = 91.128 us, max = 233.972 us, min = 5.417 us, total = 49.027 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 538 total (24 active), Execution time: mean = 81.313 s, total = 43746.486 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 444 total (0 active), Execution time: mean = 104.152 us, total = 46.244 ms, Queueing time: mean = 102.305 us, max = 205.832 us, min = 11.318 us, total = 45.424 ms InternalPubSubGcsService.grpc_server.GcsPublish - 444 total (0 active), Execution time: mean = 597.061 us, total = 265.095 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 512.656 us, total = 4.101 ms, Queueing time: mean = 72.350 us, max = 114.456 us, min = 54.388 us, total = 578.798 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:47:15,245 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8445 total (0 active) Queueing time: mean = 11.328 us, max = 2.505 ms, min = 563.000 ns, total = 95.666 ms Execution time: mean = 28.618 us, total = 241.679 ms Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 28.336 us, total = 237.995 ms, Queueing time: mean = 11.086 us, max = 2.505 ms, min = 3.043 us, total = 93.109 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:48:15,245 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 431 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 456 - cumulative published bytes: 363675 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:48:15,246 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 26273 total (4 active) Queueing time: mean = 120.418 us, max = 225.583 ms, min = -0.000 s, total = 3.164 s Execution time: mean = 244.088 us, total = 6.413 s Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 25.088 us, total = 216.734 ms, Queueing time: mean = 10.287 us, max = 6.895 ms, min = 1.268 us, total = 88.872 ms GcsInMemoryStore.Put - 3501 total (0 active), Execution time: mean = 69.711 us, total = 244.060 ms, Queueing time: mean = 493.559 us, max = 225.278 ms, min = 1.495 us, total = 1.728 s RayletLoadPulled - 2160 total (1 active), Execution time: mean = 246.808 us, total = 533.105 ms, Queueing time: mean = 74.459 us, max = 1.351 ms, min = 10.621 us, total = 160.831 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2158 total (0 active), Execution time: mean = 75.184 us, total = 162.248 ms, Queueing time: mean = 95.391 us, max = 5.512 ms, min = 11.600 us, total = 205.853 ms NodeManagerService.grpc_client.GetResourceLoad - 2158 total (0 active), Execution time: mean = 1.437 ms, total = 3.100 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 877 total (0 active), Execution time: mean = 518.627 us, total = 454.835 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 877 total (0 active), Execution time: mean = 39.141 us, total = 34.326 ms, Queueing time: mean = 90.383 us, max = 601.063 us, min = 12.201 us, total = 79.266 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 864 total (0 active), Execution time: mean = 34.908 us, total = 30.160 ms, Queueing time: mean = 104.097 us, max = 283.661 us, min = 17.215 us, total = 89.940 ms NodeInfoGcsService.grpc_server.CheckAlive - 864 total (0 active), Execution time: mean = 535.385 us, total = 462.572 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 721 total (1 active), Execution time: mean = 10.873 us, total = 7.839 ms, Queueing time: mean = 72.952 us, max = 715.636 us, min = 14.275 us, total = 52.598 ms HealthCheck - 718 total (0 active), Execution time: mean = 7.688 us, total = 5.520 ms, Queueing time: mean = 87.489 us, max = 238.751 us, min = 11.335 us, total = 62.817 ms GcsInMemoryStore.Get - 491 total (0 active), Execution time: mean = 21.787 us, total = 10.697 ms, Queueing time: mean = 13.467 us, max = 137.899 us, min = 2.227 us, total = 6.612 ms InternalKVGcsService.grpc_server.InternalKVGet - 490 total (0 active), Execution time: mean = 696.818 us, total = 341.441 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 490 total (0 active), Execution time: mean = 54.510 us, total = 26.710 ms, Queueing time: mean = 233.633 us, max = 22.982 ms, min = 9.997 us, total = 114.480 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 431 total (0 active), Execution time: mean = 623.741 us, total = 268.832 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 431 total (0 active), Execution time: mean = 89.692 us, total = 38.657 ms, Queueing time: mean = 104.450 us, max = 225.322 us, min = 17.034 us, total = 45.018 ms GCSServer.deadline_timer.debug_state_dump - 216 total (1 active), Execution time: mean = 1.292 ms, total = 279.079 ms, Queueing time: mean = 67.797 us, max = 163.420 us, min = -0.000 s, total = 14.644 ms GCSServer.deadline_timer.debug_state_event_stats_print - 36 total (1 active, 1 running), Execution time: mean = 1.536 ms, total = 55.295 ms, Queueing time: mean = 58.194 us, max = 137.733 us, min = 14.993 us, total = 2.095 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:48:15,247 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 99709 total (1 active) Queueing time: mean = 46.045 us, max = 3.642 ms, min = 2.014 us, total = 4.591 s Execution time: mean = 237.092 us, total = 23.640 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 45318 total (0 active), Execution time: mean = 491.912 us, total = 22.292 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 45318 total (0 active), Execution time: mean = 25.030 us, total = 1.134 s, Queueing time: mean = 98.846 us, max = 1.144 ms, min = 4.024 us, total = 4.480 s event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 24.187 us, total = 208.949 ms, Queueing time: mean = 8.595 us, max = 395.058 us, min = 2.014 us, total = 74.249 ms GcsTaskManager.GcJobSummary - 433 total (1 active), Execution time: mean = 10.286 us, total = 4.454 ms, Queueing time: mean = 86.136 us, max = 3.642 ms, min = 14.860 us, total = 37.297 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:48:15,247 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10710 total (25 active) Queueing time: mean = 17.248 us, max = 600.706 us, min = 2.413 us, total = 184.728 ms Execution time: mean = 4.090 s, total = 43807.298 s Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 26.888 us, total = 232.289 ms, Queueing time: mean = 9.826 us, max = 600.706 us, min = 2.413 us, total = 84.890 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 550 total (0 active), Execution time: mean = 38.678 us, total = 21.273 ms, Queueing time: mean = 91.828 us, max = 233.972 us, min = 5.417 us, total = 50.505 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 550 total (24 active), Execution time: mean = 79.649 s, total = 43806.685 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 456 total (0 active), Execution time: mean = 104.090 us, total = 47.465 ms, Queueing time: mean = 102.549 us, max = 205.832 us, min = 11.318 us, total = 46.763 ms InternalPubSubGcsService.grpc_server.GcsPublish - 456 total (0 active), Execution time: mean = 596.662 us, total = 272.078 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 512.656 us, total = 4.101 ms, Queueing time: mean = 72.350 us, max = 114.456 us, min = 54.388 us, total = 578.798 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:48:15,247 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8685 total (0 active) Queueing time: mean = 11.310 us, max = 2.505 ms, min = 563.000 ns, total = 98.225 ms Execution time: mean = 28.616 us, total = 248.534 ms Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 28.342 us, total = 244.850 ms, Queueing time: mean = 11.074 us, max = 2.505 ms, min = 3.043 us, total = 95.669 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:49:15,247 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 443 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 468 - cumulative published bytes: 373503 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:49:15,247 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 26992 total (4 active) Queueing time: mean = 118.321 us, max = 225.583 ms, min = -0.000 s, total = 3.194 s Execution time: mean = 243.879 us, total = 6.583 s Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 25.151 us, total = 223.319 ms, Queueing time: mean = 10.293 us, max = 6.895 ms, min = 1.268 us, total = 91.388 ms GcsInMemoryStore.Put - 3597 total (0 active), Execution time: mean = 67.990 us, total = 244.559 ms, Queueing time: mean = 481.650 us, max = 225.278 ms, min = 1.495 us, total = 1.732 s RayletLoadPulled - 2220 total (1 active), Execution time: mean = 247.044 us, total = 548.438 ms, Queueing time: mean = 74.655 us, max = 1.351 ms, min = 10.621 us, total = 165.735 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2218 total (0 active), Execution time: mean = 75.358 us, total = 167.144 ms, Queueing time: mean = 95.629 us, max = 5.512 ms, min = 11.600 us, total = 212.106 ms NodeManagerService.grpc_client.GetResourceLoad - 2218 total (0 active), Execution time: mean = 1.437 ms, total = 3.188 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 901 total (0 active), Execution time: mean = 518.264 us, total = 466.955 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 901 total (0 active), Execution time: mean = 39.526 us, total = 35.613 ms, Queueing time: mean = 90.285 us, max = 601.063 us, min = 12.201 us, total = 81.347 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 888 total (0 active), Execution time: mean = 35.145 us, total = 31.209 ms, Queueing time: mean = 104.600 us, max = 283.661 us, min = 17.215 us, total = 92.885 ms NodeInfoGcsService.grpc_server.CheckAlive - 888 total (0 active), Execution time: mean = 536.226 us, total = 476.169 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 741 total (1 active), Execution time: mean = 10.848 us, total = 8.038 ms, Queueing time: mean = 73.337 us, max = 715.636 us, min = 14.275 us, total = 54.342 ms HealthCheck - 738 total (0 active), Execution time: mean = 7.686 us, total = 5.672 ms, Queueing time: mean = 87.854 us, max = 238.751 us, min = 11.335 us, total = 64.837 ms GcsInMemoryStore.Get - 503 total (0 active), Execution time: mean = 21.705 us, total = 10.918 ms, Queueing time: mean = 13.428 us, max = 137.899 us, min = 2.227 us, total = 6.754 ms InternalKVGcsService.grpc_server.InternalKVGet - 502 total (0 active), Execution time: mean = 692.878 us, total = 347.825 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 502 total (0 active), Execution time: mean = 54.588 us, total = 27.403 ms, Queueing time: mean = 230.119 us, max = 22.982 ms, min = 9.997 us, total = 115.520 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 443 total (0 active), Execution time: mean = 624.745 us, total = 276.762 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 443 total (0 active), Execution time: mean = 89.835 us, total = 39.797 ms, Queueing time: mean = 104.704 us, max = 225.322 us, min = 17.034 us, total = 46.384 ms GCSServer.deadline_timer.debug_state_dump - 222 total (1 active), Execution time: mean = 1.293 ms, total = 287.151 ms, Queueing time: mean = 67.592 us, max = 163.420 us, min = -0.000 s, total = 15.005 ms GCSServer.deadline_timer.debug_state_event_stats_print - 37 total (1 active, 1 running), Execution time: mean = 1.539 ms, total = 56.925 ms, Queueing time: mean = 58.311 us, max = 137.733 us, min = 14.993 us, total = 2.157 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:49:15,248 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 102481 total (1 active) Queueing time: mean = 46.066 us, max = 3.642 ms, min = 2.014 us, total = 4.721 s Execution time: mean = 237.136 us, total = 24.302 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 46578 total (0 active), Execution time: mean = 491.977 us, total = 22.915 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 46578 total (0 active), Execution time: mean = 25.061 us, total = 1.167 s, Queueing time: mean = 98.900 us, max = 1.144 ms, min = 3.743 us, total = 4.607 s event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 24.181 us, total = 214.701 ms, Queueing time: mean = 8.599 us, max = 395.058 us, min = 2.014 us, total = 76.351 ms GcsTaskManager.GcJobSummary - 445 total (1 active), Execution time: mean = 10.299 us, total = 4.583 ms, Queueing time: mean = 85.393 us, max = 3.642 ms, min = 14.860 us, total = 38.000 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:49:15,248 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10998 total (25 active) Queueing time: mean = 17.210 us, max = 600.706 us, min = 2.413 us, total = 189.278 ms Execution time: mean = 3.989 s, total = 43867.486 s Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 26.827 us, total = 238.196 ms, Queueing time: mean = 9.816 us, max = 600.706 us, min = 2.413 us, total = 87.157 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 562 total (0 active), Execution time: mean = 38.660 us, total = 21.727 ms, Queueing time: mean = 91.901 us, max = 233.972 us, min = 5.417 us, total = 51.648 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 562 total (24 active), Execution time: mean = 78.055 s, total = 43866.858 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 468 total (0 active), Execution time: mean = 104.041 us, total = 48.691 ms, Queueing time: mean = 102.357 us, max = 205.832 us, min = 11.318 us, total = 47.903 ms InternalPubSubGcsService.grpc_server.GcsPublish - 468 total (0 active), Execution time: mean = 595.649 us, total = 278.764 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 512.656 us, total = 4.101 ms, Queueing time: mean = 72.350 us, max = 114.456 us, min = 54.388 us, total = 578.798 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:49:15,248 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8925 total (0 active) Queueing time: mean = 11.284 us, max = 2.505 ms, min = 563.000 ns, total = 100.707 ms Execution time: mean = 28.600 us, total = 255.255 ms Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 28.333 us, total = 251.570 ms, Queueing time: mean = 11.054 us, max = 2.505 ms, min = 3.043 us, total = 98.151 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:50:15,248 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 455 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 480 - cumulative published bytes: 383331 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:50:15,249 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 27711 total (4 active) Queueing time: mean = 116.308 us, max = 225.583 ms, min = -0.000 s, total = 3.223 s Execution time: mean = 243.496 us, total = 6.748 s Event stats: event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 25.160 us, total = 229.433 ms, Queueing time: mean = 10.268 us, max = 6.895 ms, min = 1.268 us, total = 93.635 ms GcsInMemoryStore.Put - 3693 total (0 active), Execution time: mean = 66.351 us, total = 245.034 ms, Queueing time: mean = 470.434 us, max = 225.278 ms, min = 1.495 us, total = 1.737 s RayletLoadPulled - 2280 total (1 active), Execution time: mean = 246.740 us, total = 562.566 ms, Queueing time: mean = 74.701 us, max = 1.351 ms, min = 10.621 us, total = 170.317 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2278 total (0 active), Execution time: mean = 75.424 us, total = 171.816 ms, Queueing time: mean = 95.660 us, max = 5.512 ms, min = 11.600 us, total = 217.914 ms NodeManagerService.grpc_client.GetResourceLoad - 2278 total (0 active), Execution time: mean = 1.438 ms, total = 3.276 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 925 total (0 active), Execution time: mean = 517.420 us, total = 478.614 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 925 total (0 active), Execution time: mean = 39.754 us, total = 36.772 ms, Queueing time: mean = 90.296 us, max = 601.063 us, min = 12.201 us, total = 83.523 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 912 total (0 active), Execution time: mean = 35.108 us, total = 32.018 ms, Queueing time: mean = 104.925 us, max = 283.661 us, min = 17.215 us, total = 95.691 ms NodeInfoGcsService.grpc_server.CheckAlive - 912 total (0 active), Execution time: mean = 536.106 us, total = 488.928 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 761 total (1 active), Execution time: mean = 10.804 us, total = 8.222 ms, Queueing time: mean = 73.667 us, max = 715.636 us, min = 14.275 us, total = 56.061 ms HealthCheck - 758 total (0 active), Execution time: mean = 7.665 us, total = 5.810 ms, Queueing time: mean = 88.463 us, max = 238.751 us, min = 11.335 us, total = 67.055 ms GcsInMemoryStore.Get - 515 total (0 active), Execution time: mean = 21.968 us, total = 11.314 ms, Queueing time: mean = 13.383 us, max = 137.899 us, min = 2.227 us, total = 6.892 ms InternalKVGcsService.grpc_server.InternalKVGet - 514 total (0 active), Execution time: mean = 690.076 us, total = 354.699 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 514 total (0 active), Execution time: mean = 54.647 us, total = 28.088 ms, Queueing time: mean = 226.796 us, max = 22.982 ms, min = 9.997 us, total = 116.573 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 455 total (0 active), Execution time: mean = 623.591 us, total = 283.734 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 455 total (0 active), Execution time: mean = 89.941 us, total = 40.923 ms, Queueing time: mean = 104.902 us, max = 225.322 us, min = 17.034 us, total = 47.730 ms GCSServer.deadline_timer.debug_state_dump - 228 total (1 active), Execution time: mean = 1.295 ms, total = 295.237 ms, Queueing time: mean = 67.284 us, max = 163.420 us, min = -0.000 s, total = 15.341 ms GCSServer.deadline_timer.debug_state_event_stats_print - 38 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 57.716 ms, Queueing time: mean = 57.683 us, max = 137.733 us, min = 14.993 us, total = 2.192 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:50:15,249 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 105253 total (1 active) Queueing time: mean = 45.928 us, max = 3.642 ms, min = 1.891 us, total = 4.834 s Execution time: mean = 236.517 us, total = 24.894 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 47838 total (0 active), Execution time: mean = 490.665 us, total = 23.472 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 47838 total (0 active), Execution time: mean = 25.021 us, total = 1.197 s, Queueing time: mean = 98.606 us, max = 1.144 ms, min = 3.743 us, total = 4.717 s event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 24.133 us, total = 220.064 ms, Queueing time: mean = 8.571 us, max = 395.058 us, min = 1.891 us, total = 78.156 ms GcsTaskManager.GcJobSummary - 457 total (1 active), Execution time: mean = 10.316 us, total = 4.714 ms, Queueing time: mean = 84.766 us, max = 3.642 ms, min = 14.860 us, total = 38.738 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:50:15,249 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11286 total (25 active) Queueing time: mean = 17.163 us, max = 600.706 us, min = 2.413 us, total = 193.706 ms Execution time: mean = 3.892 s, total = 43927.615 s Event stats: event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 26.716 us, total = 243.626 ms, Queueing time: mean = 9.775 us, max = 600.706 us, min = 2.413 us, total = 89.139 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 574 total (0 active), Execution time: mean = 38.816 us, total = 22.280 ms, Queueing time: mean = 92.077 us, max = 233.972 us, min = 5.417 us, total = 52.852 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 574 total (24 active), Execution time: mean = 76.528 s, total = 43926.973 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 480 total (0 active), Execution time: mean = 104.112 us, total = 49.974 ms, Queueing time: mean = 102.385 us, max = 214.455 us, min = 11.318 us, total = 49.145 ms InternalPubSubGcsService.grpc_server.GcsPublish - 480 total (0 active), Execution time: mean = 594.443 us, total = 285.333 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 512.656 us, total = 4.101 ms, Queueing time: mean = 72.350 us, max = 114.456 us, min = 54.388 us, total = 578.798 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:50:15,249 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9165 total (0 active) Queueing time: mean = 11.251 us, max = 2.505 ms, min = 563.000 ns, total = 103.112 ms Execution time: mean = 28.569 us, total = 261.837 ms Event stats: event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 28.309 us, total = 258.152 ms, Queueing time: mean = 11.027 us, max = 2.505 ms, min = 3.043 us, total = 100.555 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:51:15,250 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 467 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 492 - cumulative published bytes: 393159 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:51:15,251 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 28429 total (4 active) Queueing time: mean = 114.409 us, max = 225.583 ms, min = -0.000 s, total = 3.253 s Execution time: mean = 242.644 us, total = 6.898 s Event stats: event_loop_lag_probe - 9358 total (0 active), Execution time: mean = 25.190 us, total = 235.731 ms, Queueing time: mean = 10.246 us, max = 6.895 ms, min = 1.268 us, total = 95.882 ms GcsInMemoryStore.Put - 3789 total (0 active), Execution time: mean = 64.799 us, total = 245.524 ms, Queueing time: mean = 459.608 us, max = 225.278 ms, min = 1.495 us, total = 1.741 s RayletLoadPulled - 2340 total (1 active), Execution time: mean = 246.725 us, total = 577.336 ms, Queueing time: mean = 74.845 us, max = 1.351 ms, min = 10.621 us, total = 175.136 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2338 total (0 active), Execution time: mean = 75.463 us, total = 176.432 ms, Queueing time: mean = 95.877 us, max = 5.512 ms, min = 11.600 us, total = 224.160 ms NodeManagerService.grpc_client.GetResourceLoad - 2338 total (0 active), Execution time: mean = 1.433 ms, total = 3.349 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 949 total (0 active), Execution time: mean = 515.807 us, total = 489.501 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 949 total (0 active), Execution time: mean = 39.968 us, total = 37.930 ms, Queueing time: mean = 90.631 us, max = 601.063 us, min = 12.201 us, total = 86.009 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 936 total (0 active), Execution time: mean = 35.035 us, total = 32.793 ms, Queueing time: mean = 105.420 us, max = 354.546 us, min = 17.215 us, total = 98.673 ms NodeInfoGcsService.grpc_server.CheckAlive - 936 total (0 active), Execution time: mean = 534.286 us, total = 500.092 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 781 total (1 active), Execution time: mean = 10.774 us, total = 8.414 ms, Queueing time: mean = 73.520 us, max = 715.636 us, min = 14.275 us, total = 57.419 ms HealthCheck - 778 total (0 active), Execution time: mean = 7.647 us, total = 5.950 ms, Queueing time: mean = 88.771 us, max = 238.751 us, min = 11.335 us, total = 69.064 ms GcsInMemoryStore.Get - 527 total (0 active), Execution time: mean = 21.919 us, total = 11.551 ms, Queueing time: mean = 13.396 us, max = 137.899 us, min = 2.227 us, total = 7.060 ms InternalKVGcsService.grpc_server.InternalKVGet - 526 total (0 active), Execution time: mean = 685.792 us, total = 360.727 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 526 total (0 active), Execution time: mean = 54.748 us, total = 28.798 ms, Queueing time: mean = 224.095 us, max = 22.982 ms, min = 9.997 us, total = 117.874 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 467 total (0 active), Execution time: mean = 622.340 us, total = 290.633 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 467 total (0 active), Execution time: mean = 90.225 us, total = 42.135 ms, Queueing time: mean = 105.154 us, max = 225.322 us, min = 17.034 us, total = 49.107 ms GCSServer.deadline_timer.debug_state_dump - 234 total (1 active), Execution time: mean = 1.304 ms, total = 305.208 ms, Queueing time: mean = 66.980 us, max = 163.420 us, min = -0.000 s, total = 15.673 ms GCSServer.deadline_timer.debug_state_event_stats_print - 39 total (1 active, 1 running), Execution time: mean = 1.523 ms, total = 59.406 ms, Queueing time: mean = 57.809 us, max = 137.733 us, min = 14.993 us, total = 2.255 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:51:15,251 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 108025 total (1 active) Queueing time: mean = 45.378 us, max = 3.642 ms, min = 1.891 us, total = 4.902 s Execution time: mean = 234.484 us, total = 25.330 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 49098 total (0 active), Execution time: mean = 486.369 us, total = 23.880 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 49098 total (0 active), Execution time: mean = 24.850 us, total = 1.220 s, Queueing time: mean = 97.410 us, max = 1.144 ms, min = 3.743 us, total = 4.783 s event_loop_lag_probe - 9359 total (0 active), Execution time: mean = 24.092 us, total = 225.478 ms, Queueing time: mean = 8.551 us, max = 395.058 us, min = 1.891 us, total = 80.033 ms GcsTaskManager.GcJobSummary - 469 total (1 active), Execution time: mean = 10.262 us, total = 4.813 ms, Queueing time: mean = 83.716 us, max = 3.642 ms, min = 14.860 us, total = 39.263 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:51:15,251 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11574 total (25 active) Queueing time: mean = 17.040 us, max = 600.706 us, min = 2.413 us, total = 197.216 ms Execution time: mean = 3.801 s, total = 43987.808 s Event stats: event_loop_lag_probe - 9359 total (0 active), Execution time: mean = 26.606 us, total = 249.004 ms, Queueing time: mean = 9.733 us, max = 600.706 us, min = 2.413 us, total = 91.087 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 586 total (0 active), Execution time: mean = 38.961 us, total = 22.831 ms, Queueing time: mean = 91.527 us, max = 233.972 us, min = 5.417 us, total = 53.635 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 586 total (24 active), Execution time: mean = 75.063 s, total = 43987.154 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 492 total (0 active), Execution time: mean = 103.917 us, total = 51.127 ms, Queueing time: mean = 101.470 us, max = 214.455 us, min = 11.318 us, total = 49.923 ms InternalPubSubGcsService.grpc_server.GcsPublish - 492 total (0 active), Execution time: mean = 590.872 us, total = 290.709 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 512.656 us, total = 4.101 ms, Queueing time: mean = 72.350 us, max = 114.456 us, min = 54.388 us, total = 578.798 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:51:15,251 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9405 total (0 active) Queueing time: mean = 11.198 us, max = 2.505 ms, min = 563.000 ns, total = 105.313 ms Execution time: mean = 28.478 us, total = 267.834 ms Event stats: event_loop_lag_probe - 9359 total (0 active), Execution time: mean = 28.224 us, total = 264.150 ms, Queueing time: mean = 10.979 us, max = 2.505 ms, min = 3.043 us, total = 102.757 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:52:15,251 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 479 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 505 - cumulative published bytes: 403021 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:52:15,252 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 29148 total (4 active) Queueing time: mean = 112.648 us, max = 225.583 ms, min = -0.000 s, total = 3.283 s Execution time: mean = 242.034 us, total = 7.055 s Event stats: event_loop_lag_probe - 9598 total (0 active), Execution time: mean = 25.236 us, total = 242.216 ms, Queueing time: mean = 10.236 us, max = 6.895 ms, min = 1.268 us, total = 98.248 ms GcsInMemoryStore.Put - 3885 total (0 active), Execution time: mean = 63.334 us, total = 246.053 ms, Queueing time: mean = 449.364 us, max = 225.278 ms, min = 1.495 us, total = 1.746 s RayletLoadPulled - 2400 total (1 active), Execution time: mean = 246.817 us, total = 592.361 ms, Queueing time: mean = 75.103 us, max = 1.351 ms, min = 10.621 us, total = 180.247 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2398 total (0 active), Execution time: mean = 75.525 us, total = 181.108 ms, Queueing time: mean = 96.337 us, max = 5.512 ms, min = 11.600 us, total = 231.015 ms NodeManagerService.grpc_client.GetResourceLoad - 2398 total (0 active), Execution time: mean = 1.429 ms, total = 3.427 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 973 total (0 active), Execution time: mean = 515.032 us, total = 501.126 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 973 total (0 active), Execution time: mean = 39.915 us, total = 38.837 ms, Queueing time: mean = 90.775 us, max = 601.063 us, min = 12.201 us, total = 88.324 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 960 total (0 active), Execution time: mean = 34.983 us, total = 33.584 ms, Queueing time: mean = 105.798 us, max = 354.546 us, min = 17.215 us, total = 101.566 ms NodeInfoGcsService.grpc_server.CheckAlive - 960 total (0 active), Execution time: mean = 534.493 us, total = 513.114 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 801 total (1 active), Execution time: mean = 10.763 us, total = 8.621 ms, Queueing time: mean = 73.514 us, max = 715.636 us, min = 14.275 us, total = 58.885 ms HealthCheck - 798 total (0 active), Execution time: mean = 7.670 us, total = 6.121 ms, Queueing time: mean = 89.334 us, max = 238.751 us, min = 11.335 us, total = 71.289 ms GcsInMemoryStore.Get - 539 total (0 active), Execution time: mean = 21.886 us, total = 11.796 ms, Queueing time: mean = 13.400 us, max = 137.899 us, min = 2.227 us, total = 7.222 ms InternalKVGcsService.grpc_server.InternalKVGet - 538 total (0 active), Execution time: mean = 682.598 us, total = 367.237 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 538 total (0 active), Execution time: mean = 54.806 us, total = 29.486 ms, Queueing time: mean = 221.488 us, max = 22.982 ms, min = 9.997 us, total = 119.161 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 479 total (0 active), Execution time: mean = 621.910 us, total = 297.895 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 479 total (0 active), Execution time: mean = 90.150 us, total = 43.182 ms, Queueing time: mean = 105.558 us, max = 225.322 us, min = 17.034 us, total = 50.562 ms GCSServer.deadline_timer.debug_state_dump - 240 total (1 active), Execution time: mean = 1.304 ms, total = 312.955 ms, Queueing time: mean = 67.094 us, max = 163.420 us, min = -0.000 s, total = 16.102 ms GCSServer.deadline_timer.debug_state_event_stats_print - 40 total (1 active, 1 running), Execution time: mean = 1.526 ms, total = 61.039 ms, Queueing time: mean = 57.698 us, max = 137.733 us, min = 14.993 us, total = 2.308 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:52:15,253 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 110796 total (1 active) Queueing time: mean = 45.156 us, max = 3.642 ms, min = 1.891 us, total = 5.003 s Execution time: mean = 233.611 us, total = 25.883 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 50358 total (0 active), Execution time: mean = 484.542 us, total = 24.401 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 50358 total (0 active), Execution time: mean = 24.758 us, total = 1.247 s, Queueing time: mean = 96.926 us, max = 1.144 ms, min = 3.743 us, total = 4.881 s event_loop_lag_probe - 9598 total (0 active), Execution time: mean = 24.051 us, total = 230.846 ms, Queueing time: mean = 8.535 us, max = 395.058 us, min = 1.891 us, total = 81.920 ms GcsTaskManager.GcJobSummary - 481 total (1 active), Execution time: mean = 10.179 us, total = 4.896 ms, Queueing time: mean = 83.477 us, max = 3.642 ms, min = 14.860 us, total = 40.152 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:52:15,253 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11910 total (25 active) Queueing time: mean = 17.166 us, max = 600.706 us, min = 2.413 us, total = 204.451 ms Execution time: mean = 4.807 s, total = 57247.937 s Event stats: event_loop_lag_probe - 9598 total (0 active), Execution time: mean = 26.501 us, total = 254.357 ms, Queueing time: mean = 9.695 us, max = 600.706 us, min = 2.413 us, total = 93.050 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 621 total (0 active), Execution time: mean = 38.640 us, total = 23.996 ms, Queueing time: mean = 92.995 us, max = 408.208 us, min = 5.417 us, total = 57.750 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 621 total (24 active), Execution time: mean = 92.186 s, total = 57247.268 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 505 total (0 active), Execution time: mean = 103.686 us, total = 52.362 ms, Queueing time: mean = 101.033 us, max = 214.455 us, min = 11.318 us, total = 51.022 ms InternalPubSubGcsService.grpc_server.GcsPublish - 505 total (0 active), Execution time: mean = 588.400 us, total = 297.142 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 536.339 us, total = 4.827 ms, Queueing time: mean = 70.901 us, max = 114.456 us, min = 54.388 us, total = 638.108 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:52:15,253 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9645 total (0 active) Queueing time: mean = 11.141 us, max = 2.505 ms, min = 563.000 ns, total = 107.454 ms Execution time: mean = 28.389 us, total = 273.816 ms Event stats: event_loop_lag_probe - 9599 total (0 active), Execution time: mean = 28.142 us, total = 270.131 ms, Queueing time: mean = 10.928 us, max = 2.505 ms, min = 3.043 us, total = 104.898 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:53:15,253 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 491 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 518 - cumulative published bytes: 412883 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:53:15,254 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 29867 total (4 active) Queueing time: mean = 111.057 us, max = 225.583 ms, min = -0.000 s, total = 3.317 s Execution time: mean = 242.274 us, total = 7.236 s Event stats: event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 25.329 us, total = 249.190 ms, Queueing time: mean = 10.243 us, max = 6.895 ms, min = 1.268 us, total = 100.766 ms GcsInMemoryStore.Put - 3981 total (0 active), Execution time: mean = 61.952 us, total = 246.631 ms, Queueing time: mean = 439.731 us, max = 225.278 ms, min = 1.495 us, total = 1.751 s RayletLoadPulled - 2460 total (1 active), Execution time: mean = 247.419 us, total = 608.650 ms, Queueing time: mean = 75.550 us, max = 1.351 ms, min = 10.621 us, total = 185.853 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2458 total (0 active), Execution time: mean = 75.798 us, total = 186.312 ms, Queueing time: mean = 96.789 us, max = 5.512 ms, min = 11.600 us, total = 237.908 ms NodeManagerService.grpc_client.GetResourceLoad - 2458 total (0 active), Execution time: mean = 1.431 ms, total = 3.518 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 997 total (0 active), Execution time: mean = 517.505 us, total = 515.952 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 997 total (0 active), Execution time: mean = 40.377 us, total = 40.256 ms, Queueing time: mean = 91.375 us, max = 601.063 us, min = 12.201 us, total = 91.101 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 984 total (0 active), Execution time: mean = 35.046 us, total = 34.486 ms, Queueing time: mean = 106.378 us, max = 354.546 us, min = 17.215 us, total = 104.676 ms NodeInfoGcsService.grpc_server.CheckAlive - 984 total (0 active), Execution time: mean = 536.371 us, total = 527.789 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 821 total (1 active), Execution time: mean = 10.759 us, total = 8.833 ms, Queueing time: mean = 73.799 us, max = 715.636 us, min = 14.275 us, total = 60.589 ms HealthCheck - 818 total (0 active), Execution time: mean = 7.699 us, total = 6.298 ms, Queueing time: mean = 89.823 us, max = 238.751 us, min = 11.335 us, total = 73.475 ms GcsInMemoryStore.Get - 551 total (0 active), Execution time: mean = 21.921 us, total = 12.078 ms, Queueing time: mean = 13.412 us, max = 137.899 us, min = 2.227 us, total = 7.390 ms InternalKVGcsService.grpc_server.InternalKVGet - 550 total (0 active), Execution time: mean = 681.646 us, total = 374.905 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 550 total (0 active), Execution time: mean = 55.012 us, total = 30.257 ms, Queueing time: mean = 219.357 us, max = 22.982 ms, min = 9.997 us, total = 120.646 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 491 total (0 active), Execution time: mean = 624.532 us, total = 306.645 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 491 total (0 active), Execution time: mean = 90.401 us, total = 44.387 ms, Queueing time: mean = 106.368 us, max = 225.322 us, min = 17.034 us, total = 52.227 ms GCSServer.deadline_timer.debug_state_dump - 246 total (1 active), Execution time: mean = 1.307 ms, total = 321.602 ms, Queueing time: mean = 67.516 us, max = 163.420 us, min = -0.000 s, total = 16.609 ms GCSServer.deadline_timer.debug_state_event_stats_print - 41 total (1 active, 1 running), Execution time: mean = 1.530 ms, total = 62.714 ms, Queueing time: mean = 57.356 us, max = 137.733 us, min = 14.993 us, total = 2.352 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:53:15,254 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 113568 total (1 active) Queueing time: mean = 45.311 us, max = 3.642 ms, min = 1.891 us, total = 5.146 s Execution time: mean = 234.318 us, total = 26.611 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 51618 total (0 active), Execution time: mean = 486.036 us, total = 25.088 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 51618 total (0 active), Execution time: mean = 24.818 us, total = 1.281 s, Queueing time: mean = 97.265 us, max = 1.144 ms, min = 3.743 us, total = 5.021 s event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 24.059 us, total = 236.694 ms, Queueing time: mean = 8.534 us, max = 395.058 us, min = 1.891 us, total = 83.958 ms GcsTaskManager.GcJobSummary - 493 total (1 active), Execution time: mean = 10.207 us, total = 5.032 ms, Queueing time: mean = 83.776 us, max = 3.642 ms, min = 14.860 us, total = 41.301 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:53:15,255 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12202 total (25 active) Queueing time: mean = 17.161 us, max = 600.706 us, min = 2.413 us, total = 209.394 ms Execution time: mean = 4.697 s, total = 57308.152 s Event stats: event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 26.478 us, total = 260.486 ms, Queueing time: mean = 9.671 us, max = 600.706 us, min = 2.413 us, total = 95.145 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 634 total (0 active), Execution time: mean = 38.809 us, total = 24.605 ms, Queueing time: mean = 93.366 us, max = 408.208 us, min = 5.417 us, total = 59.194 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 634 total (24 active), Execution time: mean = 90.390 s, total = 57307.466 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 518 total (0 active), Execution time: mean = 104.609 us, total = 54.187 ms, Queueing time: mean = 101.208 us, max = 214.455 us, min = 11.318 us, total = 52.426 ms InternalPubSubGcsService.grpc_server.GcsPublish - 518 total (0 active), Execution time: mean = 590.701 us, total = 305.983 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 536.339 us, total = 4.827 ms, Queueing time: mean = 70.901 us, max = 114.456 us, min = 54.388 us, total = 638.108 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:53:15,255 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9884 total (0 active) Queueing time: mean = 11.108 us, max = 2.505 ms, min = 563.000 ns, total = 109.792 ms Execution time: mean = 28.368 us, total = 280.390 ms Event stats: event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 28.126 us, total = 276.705 ms, Queueing time: mean = 10.900 us, max = 2.505 ms, min = 3.043 us, total = 107.236 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:54:15,255 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 503 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 531 - cumulative published bytes: 422745 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:54:15,255 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 30586 total (4 active) Queueing time: mean = 109.466 us, max = 225.583 ms, min = -0.000 s, total = 3.348 s Execution time: mean = 242.486 us, total = 7.417 s Event stats: event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 25.401 us, total = 255.992 ms, Queueing time: mean = 10.248 us, max = 6.895 ms, min = 1.268 us, total = 103.277 ms GcsInMemoryStore.Put - 4077 total (0 active), Execution time: mean = 60.620 us, total = 247.148 ms, Queueing time: mean = 430.382 us, max = 225.278 ms, min = 1.495 us, total = 1.755 s RayletLoadPulled - 2520 total (1 active), Execution time: mean = 247.857 us, total = 624.601 ms, Queueing time: mean = 75.776 us, max = 1.351 ms, min = 10.621 us, total = 190.957 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2518 total (0 active), Execution time: mean = 75.982 us, total = 191.322 ms, Queueing time: mean = 96.967 us, max = 5.512 ms, min = 11.600 us, total = 244.163 ms NodeManagerService.grpc_client.GetResourceLoad - 2518 total (0 active), Execution time: mean = 1.434 ms, total = 3.610 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1021 total (0 active), Execution time: mean = 518.967 us, total = 529.865 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1021 total (0 active), Execution time: mean = 40.536 us, total = 41.388 ms, Queueing time: mean = 91.664 us, max = 601.063 us, min = 12.201 us, total = 93.589 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1008 total (0 active), Execution time: mean = 35.066 us, total = 35.346 ms, Queueing time: mean = 107.056 us, max = 354.546 us, min = 17.215 us, total = 107.912 ms NodeInfoGcsService.grpc_server.CheckAlive - 1008 total (0 active), Execution time: mean = 538.895 us, total = 543.206 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 841 total (1 active), Execution time: mean = 10.782 us, total = 9.068 ms, Queueing time: mean = 74.022 us, max = 715.636 us, min = 14.275 us, total = 62.252 ms HealthCheck - 838 total (0 active), Execution time: mean = 7.714 us, total = 6.464 ms, Queueing time: mean = 90.406 us, max = 288.247 us, min = 11.335 us, total = 75.761 ms GcsInMemoryStore.Get - 563 total (0 active), Execution time: mean = 21.933 us, total = 12.348 ms, Queueing time: mean = 13.411 us, max = 137.899 us, min = 2.227 us, total = 7.551 ms InternalKVGcsService.grpc_server.InternalKVGet - 562 total (0 active), Execution time: mean = 680.486 us, total = 382.433 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 562 total (0 active), Execution time: mean = 55.005 us, total = 30.913 ms, Queueing time: mean = 217.064 us, max = 22.982 ms, min = 9.997 us, total = 121.990 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 503 total (0 active), Execution time: mean = 627.160 us, total = 315.461 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 503 total (0 active), Execution time: mean = 90.593 us, total = 45.568 ms, Queueing time: mean = 106.752 us, max = 225.322 us, min = 17.034 us, total = 53.696 ms GCSServer.deadline_timer.debug_state_dump - 252 total (1 active), Execution time: mean = 1.312 ms, total = 330.696 ms, Queueing time: mean = 67.683 us, max = 163.420 us, min = -0.000 s, total = 17.056 ms GCSServer.deadline_timer.debug_state_event_stats_print - 42 total (1 active, 1 running), Execution time: mean = 1.536 ms, total = 64.515 ms, Queueing time: mean = 59.071 us, max = 137.733 us, min = 14.993 us, total = 2.481 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:54:15,256 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 116340 total (1 active) Queueing time: mean = 45.485 us, max = 3.642 ms, min = 1.891 us, total = 5.292 s Execution time: mean = 234.981 us, total = 27.338 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 52878 total (0 active), Execution time: mean = 487.431 us, total = 25.774 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 52878 total (0 active), Execution time: mean = 24.876 us, total = 1.315 s, Queueing time: mean = 97.645 us, max = 1.144 ms, min = 3.743 us, total = 5.163 s event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 24.092 us, total = 242.798 ms, Queueing time: mean = 8.539 us, max = 395.058 us, min = 1.891 us, total = 86.054 ms GcsTaskManager.GcJobSummary - 505 total (1 active), Execution time: mean = 10.181 us, total = 5.141 ms, Queueing time: mean = 83.896 us, max = 3.642 ms, min = 14.860 us, total = 42.368 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:54:15,256 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12494 total (25 active) Queueing time: mean = 17.166 us, max = 600.706 us, min = 2.413 us, total = 214.468 ms Execution time: mean = 4.592 s, total = 57368.337 s Event stats: event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 26.481 us, total = 266.874 ms, Queueing time: mean = 9.658 us, max = 600.706 us, min = 2.413 us, total = 97.333 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 647 total (0 active), Execution time: mean = 39.060 us, total = 25.272 ms, Queueing time: mean = 93.883 us, max = 408.208 us, min = 5.417 us, total = 60.742 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 647 total (24 active), Execution time: mean = 88.667 s, total = 57367.634 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 531 total (0 active), Execution time: mean = 105.178 us, total = 55.849 ms, Queueing time: mean = 101.250 us, max = 214.455 us, min = 11.318 us, total = 53.764 ms InternalPubSubGcsService.grpc_server.GcsPublish - 531 total (0 active), Execution time: mean = 592.311 us, total = 314.517 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 536.339 us, total = 4.827 ms, Queueing time: mean = 70.901 us, max = 114.456 us, min = 54.388 us, total = 638.108 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:54:15,256 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10124 total (0 active) Queueing time: mean = 11.071 us, max = 2.505 ms, min = 563.000 ns, total = 112.082 ms Execution time: mean = 28.313 us, total = 286.641 ms Event stats: event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 28.077 us, total = 282.957 ms, Queueing time: mean = 10.868 us, max = 2.505 ms, min = 3.043 us, total = 109.526 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:55:15,256 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 515 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 543 - cumulative published bytes: 432573 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:55:15,257 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 31305 total (4 active) Queueing time: mean = 107.928 us, max = 225.583 ms, min = -0.000 s, total = 3.379 s Execution time: mean = 242.658 us, total = 7.596 s Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 25.475 us, total = 262.854 ms, Queueing time: mean = 10.253 us, max = 6.895 ms, min = 1.268 us, total = 105.795 ms GcsInMemoryStore.Put - 4173 total (0 active), Execution time: mean = 59.349 us, total = 247.665 ms, Queueing time: mean = 421.509 us, max = 225.278 ms, min = 1.495 us, total = 1.759 s RayletLoadPulled - 2580 total (1 active), Execution time: mean = 248.393 us, total = 640.853 ms, Queueing time: mean = 76.062 us, max = 1.351 ms, min = 10.621 us, total = 196.239 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2578 total (0 active), Execution time: mean = 76.121 us, total = 196.239 ms, Queueing time: mean = 97.198 us, max = 5.512 ms, min = 11.600 us, total = 250.576 ms NodeManagerService.grpc_client.GetResourceLoad - 2578 total (0 active), Execution time: mean = 1.436 ms, total = 3.702 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1045 total (0 active), Execution time: mean = 520.737 us, total = 544.170 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1045 total (0 active), Execution time: mean = 40.697 us, total = 42.528 ms, Queueing time: mean = 92.130 us, max = 601.063 us, min = 12.201 us, total = 96.275 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1032 total (0 active), Execution time: mean = 35.390 us, total = 36.523 ms, Queueing time: mean = 107.068 us, max = 354.546 us, min = 17.215 us, total = 110.494 ms NodeInfoGcsService.grpc_server.CheckAlive - 1032 total (0 active), Execution time: mean = 539.935 us, total = 557.213 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 861 total (1 active), Execution time: mean = 10.776 us, total = 9.278 ms, Queueing time: mean = 74.124 us, max = 715.636 us, min = 14.275 us, total = 63.821 ms HealthCheck - 858 total (0 active), Execution time: mean = 7.737 us, total = 6.638 ms, Queueing time: mean = 90.576 us, max = 288.247 us, min = 11.335 us, total = 77.714 ms GcsInMemoryStore.Get - 575 total (0 active), Execution time: mean = 21.941 us, total = 12.616 ms, Queueing time: mean = 13.408 us, max = 137.899 us, min = 2.227 us, total = 7.710 ms InternalKVGcsService.grpc_server.InternalKVGet - 574 total (0 active), Execution time: mean = 679.394 us, total = 389.972 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 574 total (0 active), Execution time: mean = 55.269 us, total = 31.724 ms, Queueing time: mean = 214.519 us, max = 22.982 ms, min = 9.997 us, total = 123.134 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 515 total (0 active), Execution time: mean = 628.885 us, total = 323.876 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 515 total (0 active), Execution time: mean = 90.832 us, total = 46.779 ms, Queueing time: mean = 107.146 us, max = 225.322 us, min = 17.034 us, total = 55.180 ms GCSServer.deadline_timer.debug_state_dump - 258 total (1 active), Execution time: mean = 1.317 ms, total = 339.789 ms, Queueing time: mean = 67.834 us, max = 163.420 us, min = -0.000 s, total = 17.501 ms GCSServer.deadline_timer.debug_state_event_stats_print - 43 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 65.423 ms, Queueing time: mean = 58.384 us, max = 137.733 us, min = 14.993 us, total = 2.510 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:55:15,257 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 119112 total (1 active) Queueing time: mean = 45.642 us, max = 3.642 ms, min = 1.891 us, total = 5.436 s Execution time: mean = 235.598 us, total = 28.063 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 54138 total (0 active), Execution time: mean = 488.752 us, total = 26.460 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 54138 total (0 active), Execution time: mean = 24.908 us, total = 1.348 s, Queueing time: mean = 97.986 us, max = 1.144 ms, min = 3.743 us, total = 5.305 s event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 24.108 us, total = 248.748 ms, Queueing time: mean = 8.547 us, max = 395.058 us, min = 1.891 us, total = 88.187 ms GcsTaskManager.GcJobSummary - 517 total (1 active), Execution time: mean = 10.146 us, total = 5.246 ms, Queueing time: mean = 84.191 us, max = 3.642 ms, min = 14.860 us, total = 43.527 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:55:15,257 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12782 total (25 active) Queueing time: mean = 17.152 us, max = 600.706 us, min = 2.413 us, total = 219.234 ms Execution time: mean = 4.493 s, total = 57428.535 s Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 26.484 us, total = 273.264 ms, Queueing time: mean = 9.655 us, max = 600.706 us, min = 2.413 us, total = 99.623 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 659 total (0 active), Execution time: mean = 39.068 us, total = 25.746 ms, Queueing time: mean = 94.103 us, max = 408.208 us, min = 5.417 us, total = 62.014 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 659 total (24 active), Execution time: mean = 87.144 s, total = 57427.816 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 543 total (0 active), Execution time: mean = 105.367 us, total = 57.214 ms, Queueing time: mean = 101.231 us, max = 214.455 us, min = 11.318 us, total = 54.969 ms InternalPubSubGcsService.grpc_server.GcsPublish - 543 total (0 active), Execution time: mean = 593.212 us, total = 322.114 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 536.339 us, total = 4.827 ms, Queueing time: mean = 70.901 us, max = 114.456 us, min = 54.388 us, total = 638.108 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:55:15,258 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10364 total (0 active) Queueing time: mean = 11.034 us, max = 2.505 ms, min = 563.000 ns, total = 114.360 ms Execution time: mean = 28.256 us, total = 292.844 ms Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 28.025 us, total = 289.159 ms, Queueing time: mean = 10.836 us, max = 2.505 ms, min = 3.043 us, total = 111.804 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:56:15,258 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 527 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 556 - cumulative published bytes: 442435 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:56:15,259 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 32024 total (4 active) Queueing time: mean = 106.300 us, max = 225.583 ms, min = -0.000 s, total = 3.404 s Execution time: mean = 242.284 us, total = 7.759 s Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 25.487 us, total = 269.097 ms, Queueing time: mean = 10.245 us, max = 6.895 ms, min = 1.268 us, total = 108.163 ms GcsInMemoryStore.Put - 4269 total (0 active), Execution time: mean = 58.116 us, total = 248.097 ms, Queueing time: mean = 412.898 us, max = 225.278 ms, min = 1.495 us, total = 1.763 s RayletLoadPulled - 2640 total (1 active), Execution time: mean = 248.111 us, total = 655.012 ms, Queueing time: mean = 76.001 us, max = 1.351 ms, min = 10.621 us, total = 200.644 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2638 total (0 active), Execution time: mean = 76.072 us, total = 200.678 ms, Queueing time: mean = 96.917 us, max = 5.512 ms, min = 11.600 us, total = 255.666 ms NodeManagerService.grpc_client.GetResourceLoad - 2638 total (0 active), Execution time: mean = 1.436 ms, total = 3.787 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1069 total (0 active), Execution time: mean = 520.217 us, total = 556.112 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1069 total (0 active), Execution time: mean = 40.622 us, total = 43.425 ms, Queueing time: mean = 91.501 us, max = 601.063 us, min = 12.201 us, total = 97.814 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1056 total (0 active), Execution time: mean = 35.364 us, total = 37.344 ms, Queueing time: mean = 107.118 us, max = 354.546 us, min = 17.215 us, total = 113.116 ms NodeInfoGcsService.grpc_server.CheckAlive - 1056 total (0 active), Execution time: mean = 540.390 us, total = 570.652 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 881 total (1 active), Execution time: mean = 10.798 us, total = 9.513 ms, Queueing time: mean = 74.106 us, max = 715.636 us, min = 14.275 us, total = 65.288 ms HealthCheck - 878 total (0 active), Execution time: mean = 7.746 us, total = 6.801 ms, Queueing time: mean = 90.410 us, max = 288.247 us, min = 11.335 us, total = 79.380 ms GcsInMemoryStore.Get - 587 total (0 active), Execution time: mean = 21.864 us, total = 12.834 ms, Queueing time: mean = 13.341 us, max = 137.899 us, min = 2.227 us, total = 7.831 ms InternalKVGcsService.grpc_server.InternalKVGet - 586 total (0 active), Execution time: mean = 675.485 us, total = 395.834 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 586 total (0 active), Execution time: mean = 55.245 us, total = 32.373 ms, Queueing time: mean = 211.346 us, max = 22.982 ms, min = 9.997 us, total = 123.849 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 527 total (0 active), Execution time: mean = 628.618 us, total = 331.282 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 527 total (0 active), Execution time: mean = 90.517 us, total = 47.702 ms, Queueing time: mean = 107.191 us, max = 225.322 us, min = 17.034 us, total = 56.490 ms GCSServer.deadline_timer.debug_state_dump - 264 total (1 active), Execution time: mean = 1.316 ms, total = 347.450 ms, Queueing time: mean = 67.682 us, max = 163.420 us, min = -0.000 s, total = 17.868 ms GCSServer.deadline_timer.debug_state_event_stats_print - 44 total (1 active, 1 running), Execution time: mean = 1.527 ms, total = 67.170 ms, Queueing time: mean = 59.212 us, max = 137.733 us, min = 14.993 us, total = 2.605 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:56:15,259 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 121884 total (1 active) Queueing time: mean = 45.775 us, max = 3.642 ms, min = 1.891 us, total = 5.579 s Execution time: mean = 235.900 us, total = 28.752 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 55398 total (0 active), Execution time: mean = 489.398 us, total = 27.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 55398 total (0 active), Execution time: mean = 24.925 us, total = 1.381 s, Queueing time: mean = 98.284 us, max = 3.327 ms, min = 3.743 us, total = 5.445 s event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 24.111 us, total = 254.565 ms, Queueing time: mean = 8.542 us, max = 395.058 us, min = 1.891 us, total = 90.190 ms GcsTaskManager.GcJobSummary - 529 total (1 active), Execution time: mean = 10.116 us, total = 5.351 ms, Queueing time: mean = 83.871 us, max = 3.642 ms, min = 14.860 us, total = 44.368 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:56:15,259 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13074 total (25 active) Queueing time: mean = 17.135 us, max = 600.706 us, min = 2.413 us, total = 224.026 ms Execution time: mean = 4.397 s, total = 57488.664 s Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 26.452 us, total = 279.285 ms, Queueing time: mean = 9.642 us, max = 600.706 us, min = 2.413 us, total = 101.795 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 672 total (0 active), Execution time: mean = 38.952 us, total = 26.175 ms, Queueing time: mean = 94.258 us, max = 408.208 us, min = 5.417 us, total = 63.341 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 672 total (24 active), Execution time: mean = 85.548 s, total = 57487.930 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 556 total (0 active), Execution time: mean = 104.724 us, total = 58.226 ms, Queueing time: mean = 101.189 us, max = 214.455 us, min = 11.318 us, total = 56.261 ms InternalPubSubGcsService.grpc_server.GcsPublish - 556 total (0 active), Execution time: mean = 593.075 us, total = 329.750 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 536.339 us, total = 4.827 ms, Queueing time: mean = 70.901 us, max = 114.456 us, min = 54.388 us, total = 638.108 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:56:15,259 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10604 total (0 active) Queueing time: mean = 10.990 us, max = 2.505 ms, min = 563.000 ns, total = 116.537 ms Execution time: mean = 28.154 us, total = 298.548 ms Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 27.928 us, total = 294.864 ms, Queueing time: mean = 10.796 us, max = 2.505 ms, min = 2.833 us, total = 113.981 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:57:15,259 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 539 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 568 - cumulative published bytes: 452263 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:57:15,261 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 32743 total (4 active) Queueing time: mean = 104.922 us, max = 225.583 ms, min = -0.000 s, total = 3.435 s Execution time: mean = 241.625 us, total = 7.912 s Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 25.505 us, total = 275.406 ms, Queueing time: mean = 10.243 us, max = 6.895 ms, min = 1.268 us, total = 110.604 ms GcsInMemoryStore.Put - 4365 total (0 active), Execution time: mean = 56.955 us, total = 248.608 ms, Queueing time: mean = 404.703 us, max = 225.278 ms, min = 1.495 us, total = 1.767 s RayletLoadPulled - 2700 total (1 active), Execution time: mean = 247.792 us, total = 669.038 ms, Queueing time: mean = 76.196 us, max = 1.351 ms, min = 10.621 us, total = 205.730 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2698 total (0 active), Execution time: mean = 76.213 us, total = 205.621 ms, Queueing time: mean = 97.294 us, max = 5.512 ms, min = 11.600 us, total = 262.500 ms NodeManagerService.grpc_client.GetResourceLoad - 2698 total (0 active), Execution time: mean = 1.431 ms, total = 3.862 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1093 total (0 active), Execution time: mean = 519.758 us, total = 568.095 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1093 total (0 active), Execution time: mean = 40.633 us, total = 44.411 ms, Queueing time: mean = 91.711 us, max = 601.063 us, min = 12.201 us, total = 100.240 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1080 total (0 active), Execution time: mean = 35.342 us, total = 38.170 ms, Queueing time: mean = 107.701 us, max = 354.546 us, min = 17.215 us, total = 116.318 ms NodeInfoGcsService.grpc_server.CheckAlive - 1080 total (0 active), Execution time: mean = 539.882 us, total = 583.073 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 901 total (1 active), Execution time: mean = 10.777 us, total = 9.710 ms, Queueing time: mean = 74.120 us, max = 715.636 us, min = 14.275 us, total = 66.782 ms HealthCheck - 898 total (0 active), Execution time: mean = 7.763 us, total = 6.971 ms, Queueing time: mean = 91.078 us, max = 288.247 us, min = 11.335 us, total = 81.788 ms GcsInMemoryStore.Get - 599 total (0 active), Execution time: mean = 21.833 us, total = 13.078 ms, Queueing time: mean = 13.344 us, max = 137.899 us, min = 2.227 us, total = 7.993 ms InternalKVGcsService.grpc_server.InternalKVGet - 598 total (0 active), Execution time: mean = 672.414 us, total = 402.103 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 598 total (0 active), Execution time: mean = 55.260 us, total = 33.045 ms, Queueing time: mean = 209.515 us, max = 22.982 ms, min = 9.997 us, total = 125.290 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 539 total (0 active), Execution time: mean = 628.210 us, total = 338.605 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 539 total (0 active), Execution time: mean = 90.740 us, total = 48.909 ms, Queueing time: mean = 107.460 us, max = 225.322 us, min = 17.034 us, total = 57.921 ms GCSServer.deadline_timer.debug_state_dump - 270 total (1 active), Execution time: mean = 1.317 ms, total = 355.509 ms, Queueing time: mean = 68.029 us, max = 163.420 us, min = -0.000 s, total = 18.368 ms GCSServer.deadline_timer.debug_state_event_stats_print - 45 total (1 active, 1 running), Execution time: mean = 1.530 ms, total = 68.860 ms, Queueing time: mean = 58.502 us, max = 137.733 us, min = 14.993 us, total = 2.633 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:57:15,261 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 124656 total (1 active) Queueing time: mean = 45.570 us, max = 3.642 ms, min = 1.891 us, total = 5.681 s Execution time: mean = 235.095 us, total = 29.306 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 56658 total (0 active), Execution time: mean = 487.714 us, total = 27.633 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 56658 total (0 active), Execution time: mean = 24.841 us, total = 1.407 s, Queueing time: mean = 97.839 us, max = 3.327 ms, min = 3.743 us, total = 5.543 s event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 24.097 us, total = 260.199 ms, Queueing time: mean = 8.531 us, max = 395.058 us, min = 1.891 us, total = 92.118 ms GcsTaskManager.GcJobSummary - 541 total (1 active), Execution time: mean = 10.111 us, total = 5.470 ms, Queueing time: mean = 83.216 us, max = 3.642 ms, min = 14.860 us, total = 45.020 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:57:15,261 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13365 total (25 active) Queueing time: mean = 17.087 us, max = 600.706 us, min = 2.413 us, total = 228.366 ms Execution time: mean = 4.351 s, total = 58148.798 s Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 26.429 us, total = 285.383 ms, Queueing time: mean = 9.627 us, max = 600.706 us, min = 2.413 us, total = 103.956 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 685 total (0 active), Execution time: mean = 38.888 us, total = 26.638 ms, Queueing time: mean = 94.073 us, max = 408.208 us, min = 5.417 us, total = 64.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 685 total (24 active), Execution time: mean = 84.888 s, total = 58148.049 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 568 total (0 active), Execution time: mean = 104.988 us, total = 59.633 ms, Queueing time: mean = 100.784 us, max = 214.455 us, min = 11.318 us, total = 57.245 ms InternalPubSubGcsService.grpc_server.GcsPublish - 568 total (0 active), Execution time: mean = 592.120 us, total = 336.324 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 487.499 us, total = 4.875 ms, Queueing time: mean = 73.368 us, max = 114.456 us, min = 54.388 us, total = 733.683 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:57:15,261 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10844 total (0 active) Queueing time: mean = 10.951 us, max = 2.505 ms, min = 563.000 ns, total = 118.755 ms Execution time: mean = 28.101 us, total = 304.724 ms Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 27.879 us, total = 301.040 ms, Queueing time: mean = 10.761 us, max = 2.505 ms, min = 2.833 us, total = 116.199 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:58:15,261 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 551 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 581 - cumulative published bytes: 462125 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:58:15,262 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 33462 total (4 active) Queueing time: mean = 103.487 us, max = 225.583 ms, min = -0.000 s, total = 3.463 s Execution time: mean = 241.388 us, total = 8.077 s Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 25.545 us, total = 281.966 ms, Queueing time: mean = 10.243 us, max = 6.895 ms, min = 1.268 us, total = 113.061 ms GcsInMemoryStore.Put - 4461 total (0 active), Execution time: mean = 55.842 us, total = 249.109 ms, Queueing time: mean = 396.971 us, max = 225.278 ms, min = 1.495 us, total = 1.771 s RayletLoadPulled - 2760 total (1 active), Execution time: mean = 247.851 us, total = 684.067 ms, Queueing time: mean = 76.261 us, max = 1.351 ms, min = 10.621 us, total = 210.480 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2758 total (0 active), Execution time: mean = 76.165 us, total = 210.064 ms, Queueing time: mean = 97.146 us, max = 5.512 ms, min = 11.600 us, total = 267.928 ms NodeManagerService.grpc_client.GetResourceLoad - 2758 total (0 active), Execution time: mean = 1.431 ms, total = 3.947 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1117 total (0 active), Execution time: mean = 520.032 us, total = 580.876 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1117 total (0 active), Execution time: mean = 40.552 us, total = 45.296 ms, Queueing time: mean = 91.610 us, max = 601.063 us, min = 12.201 us, total = 102.329 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1104 total (0 active), Execution time: mean = 35.291 us, total = 38.962 ms, Queueing time: mean = 107.718 us, max = 354.546 us, min = 17.215 us, total = 118.921 ms NodeInfoGcsService.grpc_server.CheckAlive - 1104 total (0 active), Execution time: mean = 539.551 us, total = 595.664 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 921 total (1 active), Execution time: mean = 10.750 us, total = 9.901 ms, Queueing time: mean = 74.164 us, max = 715.636 us, min = 14.275 us, total = 68.305 ms HealthCheck - 918 total (0 active), Execution time: mean = 7.764 us, total = 7.127 ms, Queueing time: mean = 90.676 us, max = 288.247 us, min = 11.335 us, total = 83.241 ms GcsInMemoryStore.Get - 611 total (0 active), Execution time: mean = 21.774 us, total = 13.304 ms, Queueing time: mean = 13.321 us, max = 137.899 us, min = 2.227 us, total = 8.139 ms InternalKVGcsService.grpc_server.InternalKVGet - 610 total (0 active), Execution time: mean = 669.793 us, total = 408.574 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 610 total (0 active), Execution time: mean = 55.213 us, total = 33.680 ms, Queueing time: mean = 206.962 us, max = 22.982 ms, min = 9.997 us, total = 126.247 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 551 total (0 active), Execution time: mean = 628.149 us, total = 346.110 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 551 total (0 active), Execution time: mean = 90.611 us, total = 49.927 ms, Queueing time: mean = 107.160 us, max = 225.322 us, min = 17.034 us, total = 59.045 ms GCSServer.deadline_timer.debug_state_dump - 276 total (1 active), Execution time: mean = 1.321 ms, total = 364.521 ms, Queueing time: mean = 68.255 us, max = 163.420 us, min = -0.000 s, total = 18.838 ms GCSServer.deadline_timer.debug_state_event_stats_print - 46 total (1 active, 1 running), Execution time: mean = 1.541 ms, total = 70.874 ms, Queueing time: mean = 58.626 us, max = 137.733 us, min = 14.993 us, total = 2.697 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:58:15,263 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 127428 total (1 active) Queueing time: mean = 45.647 us, max = 3.642 ms, min = 1.891 us, total = 5.817 s Execution time: mean = 235.323 us, total = 29.987 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 57918 total (0 active), Execution time: mean = 488.204 us, total = 28.276 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 57918 total (0 active), Execution time: mean = 24.852 us, total = 1.439 s, Queueing time: mean = 98.012 us, max = 3.327 ms, min = 3.743 us, total = 5.677 s event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 24.100 us, total = 266.013 ms, Queueing time: mean = 8.531 us, max = 395.058 us, min = 1.891 us, total = 94.169 ms GcsTaskManager.GcJobSummary - 553 total (1 active), Execution time: mean = 10.112 us, total = 5.592 ms, Queueing time: mean = 82.874 us, max = 3.642 ms, min = 14.860 us, total = 45.829 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:58:15,263 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13657 total (25 active) Queueing time: mean = 17.082 us, max = 600.706 us, min = 2.413 us, total = 233.290 ms Execution time: mean = 4.262 s, total = 58208.964 s Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 26.382 us, total = 291.201 ms, Queueing time: mean = 9.612 us, max = 600.706 us, min = 2.413 us, total = 106.092 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 698 total (0 active), Execution time: mean = 38.965 us, total = 27.197 ms, Queueing time: mean = 94.386 us, max = 408.208 us, min = 5.417 us, total = 65.881 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 698 total (24 active), Execution time: mean = 83.393 s, total = 58208.199 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 581 total (0 active), Execution time: mean = 104.952 us, total = 60.977 ms, Queueing time: mean = 100.846 us, max = 214.455 us, min = 11.318 us, total = 58.591 ms InternalPubSubGcsService.grpc_server.GcsPublish - 581 total (0 active), Execution time: mean = 592.374 us, total = 344.169 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 487.499 us, total = 4.875 ms, Queueing time: mean = 73.368 us, max = 114.456 us, min = 54.388 us, total = 733.683 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:58:15,263 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11084 total (0 active) Queueing time: mean = 10.921 us, max = 2.505 ms, min = 563.000 ns, total = 121.050 ms Execution time: mean = 28.056 us, total = 310.972 ms Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 27.839 us, total = 307.287 ms, Queueing time: mean = 10.735 us, max = 2.505 ms, min = 2.833 us, total = 118.493 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 05:59:15,263 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 563 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 594 - cumulative published bytes: 471987 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 05:59:15,264 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 34181 total (4 active) Queueing time: mean = 102.078 us, max = 225.583 ms, min = -0.000 s, total = 3.489 s Execution time: mean = 240.911 us, total = 8.235 s Event stats: event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 25.567 us, total = 288.342 ms, Queueing time: mean = 10.230 us, max = 6.895 ms, min = 1.268 us, total = 115.370 ms GcsInMemoryStore.Put - 4557 total (0 active), Execution time: mean = 54.779 us, total = 249.626 ms, Queueing time: mean = 389.533 us, max = 225.278 ms, min = 1.495 us, total = 1.775 s RayletLoadPulled - 2820 total (1 active), Execution time: mean = 247.477 us, total = 697.884 ms, Queueing time: mean = 76.023 us, max = 1.351 ms, min = 10.621 us, total = 214.386 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2818 total (0 active), Execution time: mean = 76.079 us, total = 214.390 ms, Queueing time: mean = 96.881 us, max = 5.512 ms, min = 11.600 us, total = 273.011 ms NodeManagerService.grpc_client.GetResourceLoad - 2818 total (0 active), Execution time: mean = 1.429 ms, total = 4.027 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1141 total (0 active), Execution time: mean = 519.374 us, total = 592.605 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1141 total (0 active), Execution time: mean = 40.443 us, total = 46.146 ms, Queueing time: mean = 91.603 us, max = 601.063 us, min = 12.183 us, total = 104.519 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1128 total (0 active), Execution time: mean = 35.315 us, total = 39.835 ms, Queueing time: mean = 107.995 us, max = 354.546 us, min = 17.215 us, total = 121.819 ms NodeInfoGcsService.grpc_server.CheckAlive - 1128 total (0 active), Execution time: mean = 539.741 us, total = 608.828 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 941 total (1 active), Execution time: mean = 10.727 us, total = 10.095 ms, Queueing time: mean = 73.836 us, max = 715.636 us, min = 14.275 us, total = 69.479 ms HealthCheck - 938 total (0 active), Execution time: mean = 7.759 us, total = 7.278 ms, Queueing time: mean = 90.445 us, max = 288.247 us, min = 11.335 us, total = 84.838 ms GcsInMemoryStore.Get - 623 total (0 active), Execution time: mean = 21.742 us, total = 13.545 ms, Queueing time: mean = 13.276 us, max = 137.899 us, min = 2.227 us, total = 8.271 ms InternalKVGcsService.grpc_server.InternalKVGet - 622 total (0 active), Execution time: mean = 666.841 us, total = 414.775 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 622 total (0 active), Execution time: mean = 55.137 us, total = 34.295 ms, Queueing time: mean = 204.468 us, max = 22.982 ms, min = 9.997 us, total = 127.179 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 563 total (0 active), Execution time: mean = 628.171 us, total = 353.660 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 563 total (0 active), Execution time: mean = 90.623 us, total = 51.021 ms, Queueing time: mean = 107.006 us, max = 225.322 us, min = 17.034 us, total = 60.244 ms GCSServer.deadline_timer.debug_state_dump - 282 total (1 active), Execution time: mean = 1.320 ms, total = 372.361 ms, Queueing time: mean = 68.725 us, max = 166.240 us, min = -0.000 s, total = 19.381 ms GCSServer.deadline_timer.debug_state_event_stats_print - 47 total (1 active, 1 running), Execution time: mean = 1.543 ms, total = 72.503 ms, Queueing time: mean = 58.449 us, max = 137.733 us, min = 14.993 us, total = 2.747 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 05:59:15,265 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 130200 total (1 active) Queueing time: mean = 45.596 us, max = 4.613 ms, min = 1.891 us, total = 5.937 s Execution time: mean = 235.029 us, total = 30.601 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 59178 total (0 active), Execution time: mean = 487.593 us, total = 28.855 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 59178 total (0 active), Execution time: mean = 24.808 us, total = 1.468 s, Queueing time: mean = 97.900 us, max = 4.613 ms, min = 3.743 us, total = 5.794 s event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 24.135 us, total = 272.199 ms, Queueing time: mean = 8.545 us, max = 395.058 us, min = 1.891 us, total = 96.371 ms GcsTaskManager.GcJobSummary - 565 total (1 active), Execution time: mean = 10.085 us, total = 5.698 ms, Queueing time: mean = 82.560 us, max = 3.642 ms, min = 14.860 us, total = 46.646 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 05:59:15,265 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13949 total (25 active) Queueing time: mean = 17.030 us, max = 600.706 us, min = 2.413 us, total = 237.554 ms Execution time: mean = 4.177 s, total = 58269.104 s Event stats: event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 26.291 us, total = 296.515 ms, Queueing time: mean = 9.581 us, max = 600.706 us, min = 2.413 us, total = 108.058 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 711 total (0 active), Execution time: mean = 38.854 us, total = 27.625 ms, Queueing time: mean = 94.192 us, max = 408.208 us, min = 5.417 us, total = 66.971 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 711 total (24 active), Execution time: mean = 81.953 s, total = 58268.325 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 594 total (0 active), Execution time: mean = 104.600 us, total = 62.133 ms, Queueing time: mean = 100.675 us, max = 214.455 us, min = 11.318 us, total = 59.801 ms InternalPubSubGcsService.grpc_server.GcsPublish - 594 total (0 active), Execution time: mean = 592.128 us, total = 351.724 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 487.499 us, total = 4.875 ms, Queueing time: mean = 73.368 us, max = 114.456 us, min = 54.388 us, total = 733.683 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 05:59:15,265 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11324 total (0 active) Queueing time: mean = 10.908 us, max = 2.505 ms, min = 563.000 ns, total = 123.522 ms Execution time: mean = 28.055 us, total = 317.698 ms Event stats: event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 27.843 us, total = 314.014 ms, Queueing time: mean = 10.726 us, max = 2.505 ms, min = 2.833 us, total = 120.965 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:00:15,265 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 575 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 606 - cumulative published bytes: 481815 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:00:15,266 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 34900 total (4 active) Queueing time: mean = 100.638 us, max = 225.583 ms, min = -0.000 s, total = 3.512 s Execution time: mean = 239.855 us, total = 8.371 s Event stats: event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 25.530 us, total = 294.049 ms, Queueing time: mean = 10.206 us, max = 6.895 ms, min = 1.268 us, total = 117.555 ms GcsInMemoryStore.Put - 4653 total (0 active), Execution time: mean = 53.754 us, total = 250.117 ms, Queueing time: mean = 382.263 us, max = 225.278 ms, min = 1.495 us, total = 1.779 s RayletLoadPulled - 2880 total (1 active), Execution time: mean = 246.858 us, total = 710.950 ms, Queueing time: mean = 75.721 us, max = 1.351 ms, min = 10.621 us, total = 218.078 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2878 total (0 active), Execution time: mean = 76.097 us, total = 219.007 ms, Queueing time: mean = 96.564 us, max = 5.512 ms, min = 11.600 us, total = 277.911 ms NodeManagerService.grpc_client.GetResourceLoad - 2878 total (0 active), Execution time: mean = 1.422 ms, total = 4.093 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1165 total (0 active), Execution time: mean = 517.349 us, total = 602.712 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1165 total (0 active), Execution time: mean = 40.435 us, total = 47.107 ms, Queueing time: mean = 91.326 us, max = 601.063 us, min = 12.183 us, total = 106.395 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1152 total (0 active), Execution time: mean = 35.258 us, total = 40.617 ms, Queueing time: mean = 107.551 us, max = 354.546 us, min = 17.215 us, total = 123.899 ms NodeInfoGcsService.grpc_server.CheckAlive - 1152 total (0 active), Execution time: mean = 536.939 us, total = 618.553 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 961 total (1 active), Execution time: mean = 10.657 us, total = 10.242 ms, Queueing time: mean = 73.303 us, max = 715.636 us, min = 14.275 us, total = 70.444 ms HealthCheck - 958 total (0 active), Execution time: mean = 7.734 us, total = 7.409 ms, Queueing time: mean = 89.996 us, max = 288.247 us, min = 11.335 us, total = 86.216 ms GcsInMemoryStore.Get - 635 total (0 active), Execution time: mean = 21.676 us, total = 13.764 ms, Queueing time: mean = 13.239 us, max = 137.899 us, min = 2.227 us, total = 8.407 ms InternalKVGcsService.grpc_server.InternalKVGet - 634 total (0 active), Execution time: mean = 662.551 us, total = 420.057 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 634 total (0 active), Execution time: mean = 55.190 us, total = 34.990 ms, Queueing time: mean = 201.909 us, max = 22.982 ms, min = 9.997 us, total = 128.011 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 575 total (0 active), Execution time: mean = 626.226 us, total = 360.080 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 575 total (0 active), Execution time: mean = 90.718 us, total = 52.163 ms, Queueing time: mean = 106.791 us, max = 225.322 us, min = 17.034 us, total = 61.405 ms GCSServer.deadline_timer.debug_state_dump - 288 total (1 active), Execution time: mean = 1.323 ms, total = 380.880 ms, Queueing time: mean = 68.521 us, max = 166.240 us, min = -0.000 s, total = 19.734 ms GCSServer.deadline_timer.debug_state_event_stats_print - 48 total (1 active, 1 running), Execution time: mean = 1.552 ms, total = 74.516 ms, Queueing time: mean = 58.126 us, max = 137.733 us, min = 14.993 us, total = 2.790 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:00:15,266 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 132972 total (1 active) Queueing time: mean = 45.209 us, max = 4.613 ms, min = 1.891 us, total = 6.012 s Execution time: mean = 233.514 us, total = 31.051 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 60438 total (0 active), Execution time: mean = 484.413 us, total = 29.277 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 60438 total (0 active), Execution time: mean = 24.661 us, total = 1.490 s, Queueing time: mean = 97.059 us, max = 4.613 ms, min = 3.502 us, total = 5.866 s event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 24.106 us, total = 277.647 ms, Queueing time: mean = 8.533 us, max = 395.058 us, min = 1.891 us, total = 98.285 ms GcsTaskManager.GcJobSummary - 577 total (1 active), Execution time: mean = 10.041 us, total = 5.794 ms, Queueing time: mean = 81.806 us, max = 3.642 ms, min = 14.860 us, total = 47.202 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:00:15,266 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14237 total (25 active) Queueing time: mean = 16.934 us, max = 600.706 us, min = 2.413 us, total = 241.093 ms Execution time: mean = 4.097 s, total = 58329.288 s Event stats: event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 26.245 us, total = 302.291 ms, Queueing time: mean = 9.559 us, max = 600.706 us, min = 2.413 us, total = 110.103 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 723 total (0 active), Execution time: mean = 38.816 us, total = 28.064 ms, Queueing time: mean = 93.696 us, max = 408.208 us, min = 5.417 us, total = 67.742 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 723 total (24 active), Execution time: mean = 80.676 s, total = 58328.496 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 606 total (0 active), Execution time: mean = 104.543 us, total = 63.353 ms, Queueing time: mean = 99.872 us, max = 214.455 us, min = 11.318 us, total = 60.522 ms InternalPubSubGcsService.grpc_server.GcsPublish - 606 total (0 active), Execution time: mean = 589.123 us, total = 357.009 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 487.499 us, total = 4.875 ms, Queueing time: mean = 73.368 us, max = 114.456 us, min = 54.388 us, total = 733.683 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:00:15,266 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11564 total (0 active) Queueing time: mean = 10.850 us, max = 2.505 ms, min = 563.000 ns, total = 125.471 ms Execution time: mean = 27.936 us, total = 323.046 ms Event stats: event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 27.727 us, total = 319.362 ms, Queueing time: mean = 10.672 us, max = 2.505 ms, min = 1.873 us, total = 122.915 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:01:15,266 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 587 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 618 - cumulative published bytes: 491643 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:01:15,267 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 35618 total (4 active) Queueing time: mean = 99.440 us, max = 225.583 ms, min = -0.000 s, total = 3.542 s Execution time: mean = 239.852 us, total = 8.543 s Event stats: event_loop_lag_probe - 11757 total (0 active), Execution time: mean = 25.507 us, total = 299.883 ms, Queueing time: mean = 10.195 us, max = 6.895 ms, min = 1.268 us, total = 119.860 ms GcsInMemoryStore.Put - 4749 total (0 active), Execution time: mean = 52.788 us, total = 250.692 ms, Queueing time: mean = 375.365 us, max = 225.278 ms, min = 1.495 us, total = 1.783 s RayletLoadPulled - 2940 total (1 active), Execution time: mean = 247.016 us, total = 726.227 ms, Queueing time: mean = 75.608 us, max = 1.351 ms, min = 10.621 us, total = 222.286 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2938 total (0 active), Execution time: mean = 76.089 us, total = 223.551 ms, Queueing time: mean = 96.999 us, max = 5.512 ms, min = 11.600 us, total = 284.984 ms NodeManagerService.grpc_client.GetResourceLoad - 2938 total (0 active), Execution time: mean = 1.424 ms, total = 4.182 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1189 total (0 active), Execution time: mean = 518.333 us, total = 616.298 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1189 total (0 active), Execution time: mean = 40.406 us, total = 48.043 ms, Queueing time: mean = 91.705 us, max = 601.063 us, min = 12.183 us, total = 109.037 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1176 total (0 active), Execution time: mean = 35.372 us, total = 41.597 ms, Queueing time: mean = 107.601 us, max = 354.546 us, min = 17.215 us, total = 126.539 ms NodeInfoGcsService.grpc_server.CheckAlive - 1176 total (0 active), Execution time: mean = 537.689 us, total = 632.322 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 981 total (1 active), Execution time: mean = 10.634 us, total = 10.432 ms, Queueing time: mean = 73.304 us, max = 715.636 us, min = 14.275 us, total = 71.911 ms HealthCheck - 978 total (0 active), Execution time: mean = 7.740 us, total = 7.570 ms, Queueing time: mean = 90.178 us, max = 288.247 us, min = 11.335 us, total = 88.194 ms GcsInMemoryStore.Get - 647 total (0 active), Execution time: mean = 21.750 us, total = 14.072 ms, Queueing time: mean = 13.228 us, max = 137.899 us, min = 2.227 us, total = 8.558 ms InternalKVGcsService.grpc_server.InternalKVGet - 646 total (0 active), Execution time: mean = 660.846 us, total = 426.907 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 646 total (0 active), Execution time: mean = 55.177 us, total = 35.645 ms, Queueing time: mean = 200.151 us, max = 22.982 ms, min = 9.997 us, total = 129.297 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 587 total (0 active), Execution time: mean = 627.165 us, total = 368.146 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 587 total (0 active), Execution time: mean = 90.801 us, total = 53.300 ms, Queueing time: mean = 107.048 us, max = 225.322 us, min = 17.034 us, total = 62.837 ms GCSServer.deadline_timer.debug_state_dump - 294 total (1 active), Execution time: mean = 1.326 ms, total = 389.920 ms, Queueing time: mean = 68.525 us, max = 166.240 us, min = -0.000 s, total = 20.146 ms GCSServer.deadline_timer.debug_state_event_stats_print - 49 total (1 active, 1 running), Execution time: mean = 1.542 ms, total = 75.577 ms, Queueing time: mean = 57.531 us, max = 137.733 us, min = 14.993 us, total = 2.819 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:01:15,267 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 135743 total (1 active) Queueing time: mean = 45.322 us, max = 4.613 ms, min = 1.891 us, total = 6.152 s Execution time: mean = 233.928 us, total = 31.754 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 61698 total (0 active), Execution time: mean = 485.287 us, total = 29.941 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 61698 total (0 active), Execution time: mean = 24.679 us, total = 1.523 s, Queueing time: mean = 97.302 us, max = 4.613 ms, min = 3.502 us, total = 6.003 s event_loop_lag_probe - 11757 total (0 active), Execution time: mean = 24.183 us, total = 284.318 ms, Queueing time: mean = 8.557 us, max = 395.058 us, min = 1.891 us, total = 100.600 ms GcsTaskManager.GcJobSummary - 589 total (1 active), Execution time: mean = 10.018 us, total = 5.901 ms, Queueing time: mean = 81.797 us, max = 3.642 ms, min = 14.860 us, total = 48.178 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:01:15,268 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14525 total (25 active) Queueing time: mean = 16.949 us, max = 600.706 us, min = 2.413 us, total = 246.186 ms Execution time: mean = 4.020 s, total = 58389.514 s Event stats: event_loop_lag_probe - 11758 total (0 active), Execution time: mean = 26.268 us, total = 308.857 ms, Queueing time: mean = 9.564 us, max = 600.706 us, min = 2.413 us, total = 112.449 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 735 total (0 active), Execution time: mean = 38.861 us, total = 28.563 ms, Queueing time: mean = 93.937 us, max = 408.208 us, min = 5.417 us, total = 69.044 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 735 total (24 active), Execution time: mean = 79.440 s, total = 58388.706 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 618 total (0 active), Execution time: mean = 104.722 us, total = 64.718 ms, Queueing time: mean = 100.272 us, max = 214.455 us, min = 11.318 us, total = 61.968 ms InternalPubSubGcsService.grpc_server.GcsPublish - 618 total (0 active), Execution time: mean = 590.034 us, total = 364.641 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 487.499 us, total = 4.875 ms, Queueing time: mean = 73.368 us, max = 114.456 us, min = 54.388 us, total = 733.683 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:01:15,268 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11804 total (0 active) Queueing time: mean = 10.839 us, max = 2.505 ms, min = 563.000 ns, total = 127.945 ms Execution time: mean = 27.950 us, total = 329.923 ms Event stats: event_loop_lag_probe - 11758 total (0 active), Execution time: mean = 27.746 us, total = 326.238 ms, Queueing time: mean = 10.664 us, max = 2.505 ms, min = 1.873 us, total = 125.389 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:02:15,268 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 599 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 630 - cumulative published bytes: 501471 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:02:15,269 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 36337 total (4 active) Queueing time: mean = 97.988 us, max = 225.583 ms, min = -0.000 s, total = 3.561 s Execution time: mean = 238.584 us, total = 8.669 s Event stats: event_loop_lag_probe - 11997 total (0 active), Execution time: mean = 25.408 us, total = 304.820 ms, Queueing time: mean = 10.156 us, max = 6.895 ms, min = 1.268 us, total = 121.837 ms GcsInMemoryStore.Put - 4845 total (0 active), Execution time: mean = 51.837 us, total = 251.149 ms, Queueing time: mean = 368.657 us, max = 225.278 ms, min = 1.495 us, total = 1.786 s RayletLoadPulled - 3000 total (1 active), Execution time: mean = 246.302 us, total = 738.906 ms, Queueing time: mean = 75.086 us, max = 1.351 ms, min = 10.621 us, total = 225.257 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2998 total (0 active), Execution time: mean = 75.816 us, total = 227.295 ms, Queueing time: mean = 96.175 us, max = 5.512 ms, min = 11.600 us, total = 288.332 ms NodeManagerService.grpc_client.GetResourceLoad - 2998 total (0 active), Execution time: mean = 1.416 ms, total = 4.245 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1213 total (0 active), Execution time: mean = 515.624 us, total = 625.452 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1213 total (0 active), Execution time: mean = 40.199 us, total = 48.761 ms, Queueing time: mean = 91.154 us, max = 601.063 us, min = 12.183 us, total = 110.570 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1200 total (0 active), Execution time: mean = 35.263 us, total = 42.315 ms, Queueing time: mean = 106.911 us, max = 354.546 us, min = 17.215 us, total = 128.293 ms NodeInfoGcsService.grpc_server.CheckAlive - 1200 total (0 active), Execution time: mean = 535.351 us, total = 642.422 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1001 total (1 active), Execution time: mean = 10.590 us, total = 10.601 ms, Queueing time: mean = 72.704 us, max = 715.636 us, min = 14.275 us, total = 72.777 ms HealthCheck - 998 total (0 active), Execution time: mean = 7.701 us, total = 7.686 ms, Queueing time: mean = 89.424 us, max = 288.247 us, min = 11.335 us, total = 89.245 ms GcsInMemoryStore.Get - 659 total (0 active), Execution time: mean = 21.696 us, total = 14.297 ms, Queueing time: mean = 13.139 us, max = 137.899 us, min = 2.227 us, total = 8.658 ms InternalKVGcsService.grpc_server.InternalKVGet - 658 total (0 active), Execution time: mean = 655.797 us, total = 431.514 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 658 total (0 active), Execution time: mean = 54.875 us, total = 36.108 ms, Queueing time: mean = 197.380 us, max = 22.982 ms, min = 9.184 us, total = 129.876 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 599 total (0 active), Execution time: mean = 623.676 us, total = 373.582 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 599 total (0 active), Execution time: mean = 90.552 us, total = 54.241 ms, Queueing time: mean = 105.997 us, max = 225.322 us, min = 15.775 us, total = 63.492 ms GCSServer.deadline_timer.debug_state_dump - 300 total (1 active), Execution time: mean = 1.324 ms, total = 397.056 ms, Queueing time: mean = 68.314 us, max = 166.240 us, min = -0.000 s, total = 20.494 ms GCSServer.deadline_timer.debug_state_event_stats_print - 50 total (1 active, 1 running), Execution time: mean = 1.543 ms, total = 77.128 ms, Queueing time: mean = 57.020 us, max = 137.733 us, min = 14.993 us, total = 2.851 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:02:15,269 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 138515 total (1 active) Queueing time: mean = 44.973 us, max = 4.613 ms, min = 1.891 us, total = 6.229 s Execution time: mean = 232.538 us, total = 32.210 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 62958 total (0 active), Execution time: mean = 482.360 us, total = 30.368 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 62958 total (0 active), Execution time: mean = 24.550 us, total = 1.546 s, Queueing time: mean = 96.534 us, max = 4.613 ms, min = 2.005 us, total = 6.078 s event_loop_lag_probe - 11997 total (0 active), Execution time: mean = 24.168 us, total = 289.939 ms, Queueing time: mean = 8.554 us, max = 395.058 us, min = 1.891 us, total = 102.623 ms GcsTaskManager.GcJobSummary - 601 total (1 active), Execution time: mean = 9.955 us, total = 5.983 ms, Queueing time: mean = 81.830 us, max = 3.642 ms, min = 14.860 us, total = 49.180 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:02:15,269 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14858 total (25 active) Queueing time: mean = 16.916 us, max = 600.706 us, min = 2.413 us, total = 251.335 ms Execution time: mean = 4.822 s, total = 71649.582 s Event stats: event_loop_lag_probe - 11998 total (0 active), Execution time: mean = 26.196 us, total = 314.298 ms, Queueing time: mean = 9.536 us, max = 600.706 us, min = 2.413 us, total = 114.418 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 769 total (0 active), Execution time: mean = 37.848 us, total = 29.105 ms, Queueing time: mean = 92.607 us, max = 408.208 us, min = 5.417 us, total = 71.215 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 769 total (24 active), Execution time: mean = 93.171 s, total = 71648.761 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 630 total (0 active), Execution time: mean = 104.240 us, total = 65.671 ms, Queueing time: mean = 99.909 us, max = 214.455 us, min = 11.318 us, total = 62.943 ms InternalPubSubGcsService.grpc_server.GcsPublish - 630 total (0 active), Execution time: mean = 587.481 us, total = 370.113 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 474.382 us, total = 5.218 ms, Queueing time: mean = 69.857 us, max = 114.456 us, min = 34.746 us, total = 768.429 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:02:15,269 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12044 total (0 active) Queueing time: mean = 10.798 us, max = 2.505 ms, min = 563.000 ns, total = 130.053 ms Execution time: mean = 27.866 us, total = 335.619 ms Event stats: event_loop_lag_probe - 11998 total (0 active), Execution time: mean = 27.666 us, total = 331.935 ms, Queueing time: mean = 10.627 us, max = 2.505 ms, min = 1.873 us, total = 127.497 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:03:15,270 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 611 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 642 - cumulative published bytes: 511299 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:03:15,271 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 37056 total (4 active) Queueing time: mean = 96.649 us, max = 225.583 ms, min = -0.000 s, total = 3.581 s Execution time: mean = 237.791 us, total = 8.812 s Event stats: event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 25.325 us, total = 309.902 ms, Queueing time: mean = 10.117 us, max = 6.895 ms, min = 1.268 us, total = 123.803 ms GcsInMemoryStore.Put - 4941 total (0 active), Execution time: mean = 50.928 us, total = 251.636 ms, Queueing time: mean = 362.270 us, max = 225.278 ms, min = 1.495 us, total = 1.790 s RayletLoadPulled - 3060 total (1 active), Execution time: mean = 246.000 us, total = 752.760 ms, Queueing time: mean = 74.794 us, max = 1.351 ms, min = 10.621 us, total = 228.869 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3058 total (0 active), Execution time: mean = 75.687 us, total = 231.450 ms, Queueing time: mean = 95.702 us, max = 5.512 ms, min = 11.600 us, total = 292.656 ms NodeManagerService.grpc_client.GetResourceLoad - 3058 total (0 active), Execution time: mean = 1.411 ms, total = 4.316 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1237 total (0 active), Execution time: mean = 513.132 us, total = 634.744 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1237 total (0 active), Execution time: mean = 39.979 us, total = 49.454 ms, Queueing time: mean = 90.560 us, max = 601.063 us, min = 11.893 us, total = 112.023 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1224 total (0 active), Execution time: mean = 35.209 us, total = 43.096 ms, Queueing time: mean = 106.008 us, max = 354.546 us, min = 17.215 us, total = 129.754 ms NodeInfoGcsService.grpc_server.CheckAlive - 1224 total (0 active), Execution time: mean = 533.033 us, total = 652.432 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1021 total (1 active), Execution time: mean = 10.547 us, total = 10.769 ms, Queueing time: mean = 72.330 us, max = 715.636 us, min = 14.275 us, total = 73.849 ms HealthCheck - 1018 total (0 active), Execution time: mean = 7.670 us, total = 7.808 ms, Queueing time: mean = 88.595 us, max = 288.247 us, min = 11.335 us, total = 90.190 ms GcsInMemoryStore.Get - 671 total (0 active), Execution time: mean = 21.653 us, total = 14.529 ms, Queueing time: mean = 13.101 us, max = 137.899 us, min = 2.227 us, total = 8.790 ms InternalKVGcsService.grpc_server.InternalKVGet - 670 total (0 active), Execution time: mean = 651.762 us, total = 436.681 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 670 total (0 active), Execution time: mean = 54.799 us, total = 36.715 ms, Queueing time: mean = 195.188 us, max = 22.982 ms, min = 9.184 us, total = 130.776 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 611 total (0 active), Execution time: mean = 626.872 us, total = 383.019 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 611 total (0 active), Execution time: mean = 90.545 us, total = 55.323 ms, Queueing time: mean = 105.271 us, max = 225.322 us, min = 15.775 us, total = 64.320 ms GCSServer.deadline_timer.debug_state_dump - 306 total (1 active), Execution time: mean = 1.325 ms, total = 405.527 ms, Queueing time: mean = 67.862 us, max = 166.240 us, min = -0.000 s, total = 20.766 ms GCSServer.deadline_timer.debug_state_event_stats_print - 51 total (1 active, 1 running), Execution time: mean = 1.545 ms, total = 78.799 ms, Queueing time: mean = 56.374 us, max = 137.733 us, min = 14.993 us, total = 2.875 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:03:15,271 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 141287 total (1 active) Queueing time: mean = 44.703 us, max = 4.613 ms, min = 1.891 us, total = 6.316 s Execution time: mean = 231.401 us, total = 32.694 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 64218 total (0 active), Execution time: mean = 479.922 us, total = 30.820 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 64218 total (0 active), Execution time: mean = 24.483 us, total = 1.572 s, Queueing time: mean = 95.941 us, max = 4.613 ms, min = 2.005 us, total = 6.161 s event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 24.182 us, total = 295.918 ms, Queueing time: mean = 8.560 us, max = 395.058 us, min = 1.891 us, total = 104.750 ms GcsTaskManager.GcJobSummary - 613 total (1 active), Execution time: mean = 9.928 us, total = 6.086 ms, Queueing time: mean = 81.725 us, max = 3.642 ms, min = 14.860 us, total = 50.098 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:03:15,271 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15145 total (25 active) Queueing time: mean = 16.835 us, max = 600.706 us, min = 2.413 us, total = 254.966 ms Execution time: mean = 4.735 s, total = 71709.763 s Event stats: event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 26.156 us, total = 320.073 ms, Queueing time: mean = 9.519 us, max = 600.706 us, min = 2.413 us, total = 116.484 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 781 total (0 active), Execution time: mean = 37.730 us, total = 29.468 ms, Queueing time: mean = 92.094 us, max = 408.208 us, min = 5.417 us, total = 71.925 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 781 total (24 active), Execution time: mean = 91.817 s, total = 71708.930 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 642 total (0 active), Execution time: mean = 103.995 us, total = 66.765 ms, Queueing time: mean = 99.371 us, max = 214.455 us, min = 11.318 us, total = 63.796 ms InternalPubSubGcsService.grpc_server.GcsPublish - 642 total (0 active), Execution time: mean = 585.145 us, total = 375.663 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 474.382 us, total = 5.218 ms, Queueing time: mean = 69.857 us, max = 114.456 us, min = 34.746 us, total = 768.429 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:03:15,271 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12283 total (0 active) Queueing time: mean = 10.764 us, max = 2.505 ms, min = 563.000 ns, total = 132.216 ms Execution time: mean = 27.794 us, total = 341.390 ms Event stats: event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 27.597 us, total = 337.706 ms, Queueing time: mean = 10.596 us, max = 2.505 ms, min = 1.873 us, total = 129.660 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:04:15,272 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 623 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 654 - cumulative published bytes: 521127 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:04:15,273 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 37775 total (4 active) Queueing time: mean = 95.417 us, max = 225.583 ms, min = -0.000 s, total = 3.604 s Execution time: mean = 237.425 us, total = 8.969 s Event stats: event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 25.319 us, total = 315.911 ms, Queueing time: mean = 10.104 us, max = 6.895 ms, min = 1.268 us, total = 126.064 ms GcsInMemoryStore.Put - 5037 total (0 active), Execution time: mean = 50.059 us, total = 252.145 ms, Queueing time: mean = 356.191 us, max = 225.278 ms, min = 1.495 us, total = 1.794 s RayletLoadPulled - 3120 total (1 active), Execution time: mean = 245.963 us, total = 767.406 ms, Queueing time: mean = 74.511 us, max = 1.351 ms, min = 10.621 us, total = 232.474 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3118 total (0 active), Execution time: mean = 75.772 us, total = 236.258 ms, Queueing time: mean = 95.359 us, max = 5.512 ms, min = 11.600 us, total = 297.330 ms NodeManagerService.grpc_client.GetResourceLoad - 3118 total (0 active), Execution time: mean = 1.410 ms, total = 4.396 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1261 total (0 active), Execution time: mean = 512.421 us, total = 646.162 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1261 total (0 active), Execution time: mean = 39.934 us, total = 50.357 ms, Queueing time: mean = 90.137 us, max = 601.063 us, min = 11.893 us, total = 113.663 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1248 total (0 active), Execution time: mean = 35.553 us, total = 44.370 ms, Queueing time: mean = 105.604 us, max = 354.546 us, min = 17.215 us, total = 131.793 ms NodeInfoGcsService.grpc_server.CheckAlive - 1248 total (0 active), Execution time: mean = 533.337 us, total = 665.605 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1041 total (1 active), Execution time: mean = 10.555 us, total = 10.988 ms, Queueing time: mean = 72.051 us, max = 715.636 us, min = 14.275 us, total = 75.005 ms HealthCheck - 1038 total (0 active), Execution time: mean = 7.649 us, total = 7.940 ms, Queueing time: mean = 88.230 us, max = 288.247 us, min = 11.335 us, total = 91.583 ms GcsInMemoryStore.Get - 683 total (0 active), Execution time: mean = 21.638 us, total = 14.778 ms, Queueing time: mean = 13.042 us, max = 137.899 us, min = 2.227 us, total = 8.908 ms InternalKVGcsService.grpc_server.InternalKVGet - 682 total (0 active), Execution time: mean = 648.522 us, total = 442.292 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 682 total (0 active), Execution time: mean = 54.835 us, total = 37.397 ms, Queueing time: mean = 192.792 us, max = 22.982 ms, min = 9.184 us, total = 131.484 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 623 total (0 active), Execution time: mean = 625.486 us, total = 389.678 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 623 total (0 active), Execution time: mean = 90.486 us, total = 56.373 ms, Queueing time: mean = 104.587 us, max = 225.322 us, min = 15.775 us, total = 65.158 ms GCSServer.deadline_timer.debug_state_dump - 312 total (1 active), Execution time: mean = 1.326 ms, total = 413.856 ms, Queueing time: mean = 67.462 us, max = 166.240 us, min = -0.000 s, total = 21.048 ms GCSServer.deadline_timer.debug_state_event_stats_print - 52 total (1 active, 1 running), Execution time: mean = 1.553 ms, total = 80.737 ms, Queueing time: mean = 56.776 us, max = 137.733 us, min = 14.993 us, total = 2.952 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:04:15,273 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 144059 total (1 active) Queueing time: mean = 44.650 us, max = 4.613 ms, min = 1.891 us, total = 6.432 s Execution time: mean = 231.029 us, total = 33.282 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 65478 total (0 active), Execution time: mean = 479.131 us, total = 31.373 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 65478 total (0 active), Execution time: mean = 24.454 us, total = 1.601 s, Queueing time: mean = 95.819 us, max = 4.613 ms, min = 2.005 us, total = 6.274 s event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 24.192 us, total = 301.846 ms, Queueing time: mean = 8.558 us, max = 395.058 us, min = 1.891 us, total = 106.781 ms GcsTaskManager.GcJobSummary - 625 total (1 active), Execution time: mean = 9.936 us, total = 6.210 ms, Queueing time: mean = 82.251 us, max = 3.642 ms, min = 14.860 us, total = 51.407 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:04:15,273 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15433 total (25 active) Queueing time: mean = 16.786 us, max = 600.706 us, min = 2.413 us, total = 259.054 ms Execution time: mean = 4.650 s, total = 71769.867 s Event stats: event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 26.149 us, total = 326.262 ms, Queueing time: mean = 9.504 us, max = 600.706 us, min = 2.413 us, total = 118.588 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 793 total (0 active), Execution time: mean = 37.877 us, total = 30.036 ms, Queueing time: mean = 91.972 us, max = 408.208 us, min = 5.417 us, total = 72.934 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 793 total (24 active), Execution time: mean = 90.503 s, total = 71769.018 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 654 total (0 active), Execution time: mean = 104.110 us, total = 68.088 ms, Queueing time: mean = 99.041 us, max = 214.455 us, min = 11.318 us, total = 64.773 ms InternalPubSubGcsService.grpc_server.GcsPublish - 654 total (0 active), Execution time: mean = 585.379 us, total = 382.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 474.382 us, total = 5.218 ms, Queueing time: mean = 69.857 us, max = 114.456 us, min = 34.746 us, total = 768.429 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:04:15,273 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12523 total (0 active) Queueing time: mean = 10.740 us, max = 2.505 ms, min = 563.000 ns, total = 134.491 ms Execution time: mean = 27.765 us, total = 347.704 ms Event stats: event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 27.572 us, total = 344.020 ms, Queueing time: mean = 10.574 us, max = 2.505 ms, min = 1.873 us, total = 131.935 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:05:15,274 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 635 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 666 - cumulative published bytes: 530955 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:05:15,274 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 38494 total (4 active) Queueing time: mean = 94.436 us, max = 225.583 ms, min = -0.000 s, total = 3.635 s Execution time: mean = 237.324 us, total = 9.136 s Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 25.353 us, total = 322.419 ms, Queueing time: mean = 10.106 us, max = 6.895 ms, min = 1.268 us, total = 128.520 ms GcsInMemoryStore.Put - 5133 total (0 active), Execution time: mean = 49.236 us, total = 252.727 ms, Queueing time: mean = 350.603 us, max = 225.278 ms, min = 1.495 us, total = 1.800 s RayletLoadPulled - 3180 total (1 active), Execution time: mean = 246.187 us, total = 782.874 ms, Queueing time: mean = 74.672 us, max = 1.351 ms, min = 10.621 us, total = 237.456 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3178 total (0 active), Execution time: mean = 75.805 us, total = 240.909 ms, Queueing time: mean = 95.469 us, max = 5.512 ms, min = 11.600 us, total = 303.399 ms NodeManagerService.grpc_client.GetResourceLoad - 3178 total (0 active), Execution time: mean = 1.410 ms, total = 4.480 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1285 total (0 active), Execution time: mean = 512.982 us, total = 659.182 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1285 total (0 active), Execution time: mean = 39.982 us, total = 51.377 ms, Queueing time: mean = 90.224 us, max = 601.063 us, min = 11.893 us, total = 115.938 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1272 total (0 active), Execution time: mean = 35.561 us, total = 45.233 ms, Queueing time: mean = 105.756 us, max = 354.546 us, min = 17.215 us, total = 134.521 ms NodeInfoGcsService.grpc_server.CheckAlive - 1272 total (0 active), Execution time: mean = 533.461 us, total = 678.562 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1061 total (1 active), Execution time: mean = 10.548 us, total = 11.192 ms, Queueing time: mean = 72.173 us, max = 715.636 us, min = 14.275 us, total = 76.576 ms HealthCheck - 1058 total (0 active), Execution time: mean = 7.659 us, total = 8.103 ms, Queueing time: mean = 88.461 us, max = 288.247 us, min = 11.335 us, total = 93.592 ms GcsInMemoryStore.Get - 695 total (0 active), Execution time: mean = 21.621 us, total = 15.027 ms, Queueing time: mean = 13.035 us, max = 137.899 us, min = 2.227 us, total = 9.059 ms InternalKVGcsService.grpc_server.InternalKVGet - 694 total (0 active), Execution time: mean = 647.397 us, total = 449.293 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 694 total (0 active), Execution time: mean = 54.982 us, total = 38.157 ms, Queueing time: mean = 191.280 us, max = 22.982 ms, min = 9.184 us, total = 132.748 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 635 total (0 active), Execution time: mean = 626.470 us, total = 397.809 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 635 total (0 active), Execution time: mean = 90.628 us, total = 57.549 ms, Queueing time: mean = 104.708 us, max = 225.322 us, min = 15.775 us, total = 66.490 ms GCSServer.deadline_timer.debug_state_dump - 318 total (1 active), Execution time: mean = 1.327 ms, total = 422.023 ms, Queueing time: mean = 67.578 us, max = 166.240 us, min = -0.000 s, total = 21.490 ms GCSServer.deadline_timer.debug_state_event_stats_print - 53 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 82.631 ms, Queueing time: mean = 56.974 us, max = 137.733 us, min = 14.993 us, total = 3.020 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:05:15,274 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 146831 total (1 active) Queueing time: mean = 44.637 us, max = 4.613 ms, min = 1.891 us, total = 6.554 s Execution time: mean = 230.893 us, total = 33.902 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 66738 total (0 active), Execution time: mean = 478.839 us, total = 31.957 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 66738 total (0 active), Execution time: mean = 24.443 us, total = 1.631 s, Queueing time: mean = 95.788 us, max = 4.613 ms, min = 2.005 us, total = 6.393 s event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 24.212 us, total = 307.902 ms, Queueing time: mean = 8.566 us, max = 395.058 us, min = 1.891 us, total = 108.938 ms GcsTaskManager.GcJobSummary - 637 total (1 active), Execution time: mean = 9.910 us, total = 6.313 ms, Queueing time: mean = 82.320 us, max = 3.642 ms, min = 14.860 us, total = 52.438 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:05:15,274 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15721 total (25 active) Queueing time: mean = 16.769 us, max = 600.706 us, min = 2.413 us, total = 263.632 ms Execution time: mean = 4.569 s, total = 71830.029 s Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 26.134 us, total = 332.349 ms, Queueing time: mean = 9.501 us, max = 600.706 us, min = 2.413 us, total = 120.819 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 805 total (0 active), Execution time: mean = 38.015 us, total = 30.602 ms, Queueing time: mean = 91.969 us, max = 408.208 us, min = 5.417 us, total = 74.035 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 805 total (24 active), Execution time: mean = 89.229 s, total = 71829.165 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 666 total (0 active), Execution time: mean = 104.418 us, total = 69.542 ms, Queueing time: mean = 99.126 us, max = 214.455 us, min = 11.318 us, total = 66.018 ms InternalPubSubGcsService.grpc_server.GcsPublish - 666 total (0 active), Execution time: mean = 585.724 us, total = 390.092 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 474.382 us, total = 5.218 ms, Queueing time: mean = 69.857 us, max = 114.456 us, min = 34.746 us, total = 768.429 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:05:15,274 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12763 total (0 active) Queueing time: mean = 10.730 us, max = 2.505 ms, min = 563.000 ns, total = 136.948 ms Execution time: mean = 27.767 us, total = 354.389 ms Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 27.578 us, total = 350.704 ms, Queueing time: mean = 10.568 us, max = 2.505 ms, min = 1.873 us, total = 134.392 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:06:15,275 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 647 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 679 - cumulative published bytes: 540817 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:06:15,276 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 39213 total (4 active) Queueing time: mean = 93.474 us, max = 225.583 ms, min = -0.000 s, total = 3.665 s Execution time: mean = 237.355 us, total = 9.307 s Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 25.413 us, total = 329.281 ms, Queueing time: mean = 10.119 us, max = 6.895 ms, min = 1.268 us, total = 131.112 ms GcsInMemoryStore.Put - 5229 total (0 active), Execution time: mean = 48.430 us, total = 253.242 ms, Queueing time: mean = 345.088 us, max = 225.278 ms, min = 1.495 us, total = 1.804 s RayletLoadPulled - 3240 total (1 active), Execution time: mean = 246.597 us, total = 798.975 ms, Queueing time: mean = 74.668 us, max = 1.351 ms, min = 10.621 us, total = 241.924 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3238 total (0 active), Execution time: mean = 75.834 us, total = 245.549 ms, Queueing time: mean = 95.629 us, max = 5.512 ms, min = 11.600 us, total = 309.648 ms NodeManagerService.grpc_client.GetResourceLoad - 3238 total (0 active), Execution time: mean = 1.411 ms, total = 4.567 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1309 total (0 active), Execution time: mean = 513.713 us, total = 672.450 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1309 total (0 active), Execution time: mean = 40.091 us, total = 52.479 ms, Queueing time: mean = 90.316 us, max = 601.063 us, min = 11.893 us, total = 118.224 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1296 total (0 active), Execution time: mean = 35.502 us, total = 46.010 ms, Queueing time: mean = 105.749 us, max = 354.546 us, min = 17.215 us, total = 137.051 ms NodeInfoGcsService.grpc_server.CheckAlive - 1296 total (0 active), Execution time: mean = 533.771 us, total = 691.767 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1081 total (1 active), Execution time: mean = 10.551 us, total = 11.405 ms, Queueing time: mean = 72.550 us, max = 715.636 us, min = 14.275 us, total = 78.427 ms HealthCheck - 1078 total (0 active), Execution time: mean = 7.670 us, total = 8.268 ms, Queueing time: mean = 88.602 us, max = 288.247 us, min = 11.335 us, total = 95.513 ms GcsInMemoryStore.Get - 707 total (0 active), Execution time: mean = 21.620 us, total = 15.285 ms, Queueing time: mean = 13.039 us, max = 137.899 us, min = 2.227 us, total = 9.219 ms InternalKVGcsService.grpc_server.InternalKVGet - 706 total (0 active), Execution time: mean = 646.272 us, total = 456.268 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 706 total (0 active), Execution time: mean = 55.242 us, total = 39.001 ms, Queueing time: mean = 189.836 us, max = 22.982 ms, min = 9.184 us, total = 134.024 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 647 total (0 active), Execution time: mean = 627.660 us, total = 406.096 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 647 total (0 active), Execution time: mean = 90.799 us, total = 58.747 ms, Queueing time: mean = 105.003 us, max = 225.322 us, min = 15.775 us, total = 67.937 ms GCSServer.deadline_timer.debug_state_dump - 324 total (1 active), Execution time: mean = 1.330 ms, total = 430.822 ms, Queueing time: mean = 67.930 us, max = 166.240 us, min = -0.000 s, total = 22.009 ms GCSServer.deadline_timer.debug_state_event_stats_print - 54 total (1 active, 1 running), Execution time: mean = 1.550 ms, total = 83.695 ms, Queueing time: mean = 56.629 us, max = 137.733 us, min = 14.993 us, total = 3.058 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:06:15,276 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 149603 total (1 active) Queueing time: mean = 44.735 us, max = 4.613 ms, min = 1.891 us, total = 6.693 s Execution time: mean = 231.160 us, total = 34.582 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 67998 total (0 active), Execution time: mean = 479.405 us, total = 32.599 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 67998 total (0 active), Execution time: mean = 24.458 us, total = 1.663 s, Queueing time: mean = 96.001 us, max = 4.613 ms, min = 2.005 us, total = 6.528 s event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 24.242 us, total = 314.100 ms, Queueing time: mean = 8.580 us, max = 395.058 us, min = 1.891 us, total = 111.168 ms GcsTaskManager.GcJobSummary - 649 total (1 active), Execution time: mean = 9.917 us, total = 6.436 ms, Queueing time: mean = 82.449 us, max = 3.642 ms, min = 14.860 us, total = 53.510 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:06:15,276 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16013 total (25 active) Queueing time: mean = 16.766 us, max = 600.706 us, min = 2.413 us, total = 268.479 ms Execution time: mean = 4.489 s, total = 71890.235 s Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 26.147 us, total = 338.793 ms, Queueing time: mean = 9.505 us, max = 600.706 us, min = 2.413 us, total = 123.151 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 818 total (0 active), Execution time: mean = 38.178 us, total = 31.229 ms, Queueing time: mean = 91.966 us, max = 408.208 us, min = 5.417 us, total = 75.229 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 818 total (24 active), Execution time: mean = 87.884 s, total = 71889.355 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 679 total (0 active), Execution time: mean = 104.509 us, total = 70.962 ms, Queueing time: mean = 99.176 us, max = 214.455 us, min = 11.318 us, total = 67.340 ms InternalPubSubGcsService.grpc_server.GcsPublish - 679 total (0 active), Execution time: mean = 585.894 us, total = 397.822 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 474.382 us, total = 5.218 ms, Queueing time: mean = 69.857 us, max = 114.456 us, min = 34.746 us, total = 768.429 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:06:15,276 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13003 total (0 active) Queueing time: mean = 10.715 us, max = 2.505 ms, min = 563.000 ns, total = 139.325 ms Execution time: mean = 27.746 us, total = 360.777 ms Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 27.560 us, total = 357.092 ms, Queueing time: mean = 10.556 us, max = 2.505 ms, min = 1.873 us, total = 136.769 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:07:15,277 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 659 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 691 - cumulative published bytes: 550645 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:07:15,277 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 39932 total (4 active) Queueing time: mean = 92.440 us, max = 225.583 ms, min = -0.000 s, total = 3.691 s Execution time: mean = 236.302 us, total = 9.436 s Event stats: event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 25.419 us, total = 335.453 ms, Queueing time: mean = 10.114 us, max = 6.895 ms, min = 1.268 us, total = 133.472 ms GcsInMemoryStore.Put - 5325 total (0 active), Execution time: mean = 47.651 us, total = 253.739 ms, Queueing time: mean = 339.741 us, max = 225.278 ms, min = 1.495 us, total = 1.809 s RayletLoadPulled - 3300 total (1 active), Execution time: mean = 246.397 us, total = 813.110 ms, Queueing time: mean = 74.512 us, max = 1.351 ms, min = 10.621 us, total = 245.889 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3298 total (0 active), Execution time: mean = 75.758 us, total = 249.850 ms, Queueing time: mean = 95.500 us, max = 5.512 ms, min = 11.600 us, total = 314.960 ms NodeManagerService.grpc_client.GetResourceLoad - 3298 total (0 active), Execution time: mean = 1.403 ms, total = 4.627 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1333 total (0 active), Execution time: mean = 510.823 us, total = 680.927 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1333 total (0 active), Execution time: mean = 40.173 us, total = 53.551 ms, Queueing time: mean = 90.078 us, max = 601.063 us, min = 11.893 us, total = 120.074 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1320 total (0 active), Execution time: mean = 35.431 us, total = 46.769 ms, Queueing time: mean = 105.772 us, max = 354.546 us, min = 17.215 us, total = 139.619 ms NodeInfoGcsService.grpc_server.CheckAlive - 1320 total (0 active), Execution time: mean = 532.000 us, total = 702.239 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1101 total (1 active), Execution time: mean = 10.506 us, total = 11.567 ms, Queueing time: mean = 72.361 us, max = 715.636 us, min = 14.275 us, total = 79.669 ms HealthCheck - 1098 total (0 active), Execution time: mean = 7.662 us, total = 8.412 ms, Queueing time: mean = 88.567 us, max = 288.247 us, min = 11.335 us, total = 97.246 ms GcsInMemoryStore.Get - 719 total (0 active), Execution time: mean = 21.535 us, total = 15.483 ms, Queueing time: mean = 13.010 us, max = 137.899 us, min = 2.227 us, total = 9.354 ms InternalKVGcsService.grpc_server.InternalKVGet - 718 total (0 active), Execution time: mean = 641.848 us, total = 460.847 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 718 total (0 active), Execution time: mean = 55.451 us, total = 39.814 ms, Queueing time: mean = 187.765 us, max = 22.982 ms, min = 9.184 us, total = 134.816 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 659 total (0 active), Execution time: mean = 624.549 us, total = 411.578 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 659 total (0 active), Execution time: mean = 90.534 us, total = 59.662 ms, Queueing time: mean = 104.341 us, max = 225.322 us, min = 15.775 us, total = 68.761 ms GCSServer.deadline_timer.debug_state_dump - 330 total (1 active), Execution time: mean = 1.332 ms, total = 439.441 ms, Queueing time: mean = 67.908 us, max = 166.240 us, min = -0.000 s, total = 22.410 ms GCSServer.deadline_timer.debug_state_event_stats_print - 55 total (1 active, 1 running), Execution time: mean = 1.554 ms, total = 85.465 ms, Queueing time: mean = 57.273 us, max = 137.733 us, min = 14.993 us, total = 3.150 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:07:15,278 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 152375 total (1 active) Queueing time: mean = 44.286 us, max = 4.613 ms, min = 1.891 us, total = 6.748 s Execution time: mean = 229.375 us, total = 34.951 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 69258 total (0 active), Execution time: mean = 475.652 us, total = 32.943 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 69258 total (0 active), Execution time: mean = 24.283 us, total = 1.682 s, Queueing time: mean = 95.018 us, max = 4.613 ms, min = 2.005 us, total = 6.581 s event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 24.247 us, total = 319.985 ms, Queueing time: mean = 8.589 us, max = 395.058 us, min = 1.891 us, total = 113.350 ms GcsTaskManager.GcJobSummary - 661 total (1 active), Execution time: mean = 9.874 us, total = 6.527 ms, Queueing time: mean = 81.620 us, max = 3.642 ms, min = 14.860 us, total = 53.951 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:07:15,278 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16304 total (25 active) Queueing time: mean = 16.672 us, max = 600.706 us, min = 2.413 us, total = 271.821 ms Execution time: mean = 4.450 s, total = 72550.441 s Event stats: event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 26.146 us, total = 345.048 ms, Queueing time: mean = 9.499 us, max = 600.706 us, min = 2.413 us, total = 125.353 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 831 total (0 active), Execution time: mean = 38.015 us, total = 31.591 ms, Queueing time: mean = 91.063 us, max = 408.208 us, min = 5.417 us, total = 75.673 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 831 total (24 active), Execution time: mean = 87.304 s, total = 72549.549 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 691 total (0 active), Execution time: mean = 104.443 us, total = 72.170 ms, Queueing time: mean = 98.369 us, max = 214.455 us, min = 11.318 us, total = 67.973 ms InternalPubSubGcsService.grpc_server.GcsPublish - 691 total (0 active), Execution time: mean = 582.505 us, total = 402.511 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 437.940 us, total = 5.255 ms, Queueing time: mean = 69.264 us, max = 114.456 us, min = 34.746 us, total = 831.168 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:07:15,278 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13243 total (0 active) Queueing time: mean = 10.687 us, max = 2.505 ms, min = 563.000 ns, total = 141.525 ms Execution time: mean = 27.686 us, total = 366.646 ms Event stats: event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 27.503 us, total = 362.962 ms, Queueing time: mean = 10.530 us, max = 2.505 ms, min = 1.873 us, total = 138.969 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:08:15,278 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 671 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 703 - cumulative published bytes: 560473 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:08:15,279 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 40651 total (4 active) Queueing time: mean = 91.539 us, max = 225.583 ms, min = -0.000 s, total = 3.721 s Execution time: mean = 235.012 us, total = 9.553 s Event stats: event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 25.420 us, total = 341.563 ms, Queueing time: mean = 10.112 us, max = 6.895 ms, min = 1.268 us, total = 135.875 ms GcsInMemoryStore.Put - 5421 total (0 active), Execution time: mean = 46.900 us, total = 254.242 ms, Queueing time: mean = 334.727 us, max = 225.278 ms, min = 1.495 us, total = 1.815 s RayletLoadPulled - 3360 total (1 active), Execution time: mean = 245.910 us, total = 826.259 ms, Queueing time: mean = 74.447 us, max = 1.351 ms, min = 10.621 us, total = 250.142 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3358 total (0 active), Execution time: mean = 75.638 us, total = 253.994 ms, Queueing time: mean = 95.562 us, max = 5.512 ms, min = 11.600 us, total = 320.896 ms NodeManagerService.grpc_client.GetResourceLoad - 3358 total (0 active), Execution time: mean = 1.393 ms, total = 4.678 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1357 total (0 active), Execution time: mean = 508.212 us, total = 689.644 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1357 total (0 active), Execution time: mean = 40.141 us, total = 54.472 ms, Queueing time: mean = 90.208 us, max = 601.063 us, min = 11.893 us, total = 122.412 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1344 total (0 active), Execution time: mean = 35.298 us, total = 47.441 ms, Queueing time: mean = 106.053 us, max = 354.546 us, min = 17.215 us, total = 142.535 ms NodeInfoGcsService.grpc_server.CheckAlive - 1344 total (0 active), Execution time: mean = 529.668 us, total = 711.874 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1121 total (1 active), Execution time: mean = 10.514 us, total = 11.786 ms, Queueing time: mean = 72.195 us, max = 715.636 us, min = 14.275 us, total = 80.930 ms HealthCheck - 1118 total (0 active), Execution time: mean = 7.656 us, total = 8.560 ms, Queueing time: mean = 88.900 us, max = 288.247 us, min = 11.335 us, total = 99.390 ms GcsInMemoryStore.Get - 731 total (0 active), Execution time: mean = 21.455 us, total = 15.684 ms, Queueing time: mean = 12.996 us, max = 137.899 us, min = 2.227 us, total = 9.500 ms InternalKVGcsService.grpc_server.InternalKVGet - 730 total (0 active), Execution time: mean = 637.898 us, total = 465.666 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 730 total (0 active), Execution time: mean = 55.515 us, total = 40.526 ms, Queueing time: mean = 186.225 us, max = 22.982 ms, min = 9.184 us, total = 135.944 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 671 total (0 active), Execution time: mean = 622.127 us, total = 417.447 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 671 total (0 active), Execution time: mean = 90.353 us, total = 60.627 ms, Queueing time: mean = 104.601 us, max = 225.322 us, min = 15.775 us, total = 70.187 ms GCSServer.deadline_timer.debug_state_dump - 336 total (1 active), Execution time: mean = 1.332 ms, total = 447.679 ms, Queueing time: mean = 67.875 us, max = 166.240 us, min = -0.000 s, total = 22.806 ms GCSServer.deadline_timer.debug_state_event_stats_print - 56 total (1 active, 1 running), Execution time: mean = 1.555 ms, total = 87.061 ms, Queueing time: mean = 57.123 us, max = 137.733 us, min = 14.993 us, total = 3.199 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:08:15,279 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 155147 total (1 active) Queueing time: mean = 43.674 us, max = 4.613 ms, min = 1.891 us, total = 6.776 s Execution time: mean = 227.049 us, total = 35.226 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 70518 total (0 active), Execution time: mean = 470.773 us, total = 33.198 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 70518 total (0 active), Execution time: mean = 24.055 us, total = 1.696 s, Queueing time: mean = 93.682 us, max = 4.613 ms, min = 2.005 us, total = 6.606 s event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 24.193 us, total = 325.077 ms, Queueing time: mean = 8.574 us, max = 395.058 us, min = 1.891 us, total = 115.213 ms GcsTaskManager.GcJobSummary - 673 total (1 active), Execution time: mean = 9.822 us, total = 6.610 ms, Queueing time: mean = 80.797 us, max = 3.642 ms, min = 14.860 us, total = 54.377 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:08:15,280 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16592 total (25 active) Queueing time: mean = 16.524 us, max = 600.706 us, min = 2.413 us, total = 274.162 ms Execution time: mean = 4.376 s, total = 72610.495 s Event stats: event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 26.032 us, total = 349.797 ms, Queueing time: mean = 9.460 us, max = 600.706 us, min = 2.413 us, total = 127.112 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 843 total (0 active), Execution time: mean = 37.881 us, total = 31.933 ms, Queueing time: mean = 90.130 us, max = 408.208 us, min = 5.417 us, total = 75.979 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 843 total (24 active), Execution time: mean = 86.132 s, total = 72609.593 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 703 total (0 active), Execution time: mean = 103.989 us, total = 73.104 ms, Queueing time: mean = 97.082 us, max = 214.455 us, min = 11.318 us, total = 68.249 ms InternalPubSubGcsService.grpc_server.GcsPublish - 703 total (0 active), Execution time: mean = 577.639 us, total = 406.081 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 437.940 us, total = 5.255 ms, Queueing time: mean = 69.264 us, max = 114.456 us, min = 34.746 us, total = 831.168 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:08:15,280 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13483 total (0 active) Queueing time: mean = 10.637 us, max = 2.505 ms, min = 563.000 ns, total = 143.415 ms Execution time: mean = 27.572 us, total = 371.750 ms Event stats: event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 27.392 us, total = 368.066 ms, Queueing time: mean = 10.483 us, max = 2.505 ms, min = 1.873 us, total = 140.859 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:09:15,280 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 683 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 715 - cumulative published bytes: 570301 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:09:15,281 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 41370 total (4 active) Queueing time: mean = 90.460 us, max = 225.583 ms, min = -0.000 s, total = 3.742 s Execution time: mean = 233.726 us, total = 9.669 s Event stats: event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 25.386 us, total = 347.198 ms, Queueing time: mean = 10.090 us, max = 6.895 ms, min = 1.268 us, total = 138.001 ms GcsInMemoryStore.Put - 5517 total (0 active), Execution time: mean = 46.157 us, total = 254.649 ms, Queueing time: mean = 329.565 us, max = 225.278 ms, min = 1.495 us, total = 1.818 s RayletLoadPulled - 3420 total (1 active), Execution time: mean = 245.218 us, total = 838.647 ms, Queueing time: mean = 74.004 us, max = 1.351 ms, min = 10.621 us, total = 253.093 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3418 total (0 active), Execution time: mean = 75.373 us, total = 257.625 ms, Queueing time: mean = 94.983 us, max = 5.512 ms, min = 11.600 us, total = 324.650 ms NodeManagerService.grpc_client.GetResourceLoad - 3418 total (0 active), Execution time: mean = 1.384 ms, total = 4.731 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1381 total (0 active), Execution time: mean = 505.208 us, total = 697.692 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1381 total (0 active), Execution time: mean = 40.167 us, total = 55.470 ms, Queueing time: mean = 89.882 us, max = 601.063 us, min = 11.893 us, total = 124.128 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1368 total (0 active), Execution time: mean = 35.211 us, total = 48.169 ms, Queueing time: mean = 105.865 us, max = 354.546 us, min = 17.215 us, total = 144.824 ms NodeInfoGcsService.grpc_server.CheckAlive - 1368 total (0 active), Execution time: mean = 527.217 us, total = 721.233 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1141 total (1 active), Execution time: mean = 10.450 us, total = 11.924 ms, Queueing time: mean = 71.747 us, max = 715.636 us, min = 13.542 us, total = 81.863 ms HealthCheck - 1138 total (0 active), Execution time: mean = 7.633 us, total = 8.686 ms, Queueing time: mean = 88.467 us, max = 288.247 us, min = 11.335 us, total = 100.675 ms GcsInMemoryStore.Get - 743 total (0 active), Execution time: mean = 21.388 us, total = 15.891 ms, Queueing time: mean = 12.951 us, max = 137.899 us, min = 2.227 us, total = 9.623 ms InternalKVGcsService.grpc_server.InternalKVGet - 742 total (0 active), Execution time: mean = 633.456 us, total = 470.024 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 742 total (0 active), Execution time: mean = 55.500 us, total = 41.181 ms, Queueing time: mean = 184.362 us, max = 22.982 ms, min = 9.184 us, total = 136.796 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 683 total (0 active), Execution time: mean = 618.945 us, total = 422.740 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 683 total (0 active), Execution time: mean = 90.308 us, total = 61.681 ms, Queueing time: mean = 104.329 us, max = 225.322 us, min = 15.775 us, total = 71.257 ms GCSServer.deadline_timer.debug_state_dump - 342 total (1 active), Execution time: mean = 1.332 ms, total = 455.715 ms, Queueing time: mean = 67.736 us, max = 166.240 us, min = -0.000 s, total = 23.166 ms GCSServer.deadline_timer.debug_state_event_stats_print - 57 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 88.801 ms, Queueing time: mean = 57.267 us, max = 137.733 us, min = 14.993 us, total = 3.264 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:09:15,281 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 157919 total (1 active) Queueing time: mean = 43.163 us, max = 4.613 ms, min = 1.891 us, total = 6.816 s Execution time: mean = 225.086 us, total = 35.545 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 71778 total (0 active), Execution time: mean = 466.652 us, total = 33.495 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 71778 total (0 active), Execution time: mean = 23.863 us, total = 1.713 s, Queueing time: mean = 92.567 us, max = 4.613 ms, min = 2.005 us, total = 6.644 s event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 24.162 us, total = 330.465 ms, Queueing time: mean = 8.567 us, max = 395.058 us, min = 1.891 us, total = 117.174 ms GcsTaskManager.GcJobSummary - 685 total (1 active), Execution time: mean = 9.773 us, total = 6.694 ms, Queueing time: mean = 80.106 us, max = 3.642 ms, min = 14.860 us, total = 54.873 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:09:15,281 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16880 total (25 active) Queueing time: mean = 16.385 us, max = 600.706 us, min = 2.413 us, total = 276.582 ms Execution time: mean = 4.305 s, total = 72670.643 s Event stats: event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 25.958 us, total = 355.027 ms, Queueing time: mean = 9.434 us, max = 600.706 us, min = 2.413 us, total = 129.031 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 855 total (0 active), Execution time: mean = 37.663 us, total = 32.202 ms, Queueing time: mean = 89.102 us, max = 408.208 us, min = 5.417 us, total = 76.182 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 855 total (24 active), Execution time: mean = 84.994 s, total = 72669.731 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 715 total (0 active), Execution time: mean = 103.383 us, total = 73.919 ms, Queueing time: mean = 95.870 us, max = 214.455 us, min = 11.318 us, total = 68.547 ms InternalPubSubGcsService.grpc_server.GcsPublish - 715 total (0 active), Execution time: mean = 572.718 us, total = 409.493 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 437.940 us, total = 5.255 ms, Queueing time: mean = 69.264 us, max = 114.456 us, min = 34.746 us, total = 831.168 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:09:15,281 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13723 total (0 active) Queueing time: mean = 10.584 us, max = 2.505 ms, min = 563.000 ns, total = 145.239 ms Execution time: mean = 27.442 us, total = 376.582 ms Event stats: event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 27.265 us, total = 372.898 ms, Queueing time: mean = 10.432 us, max = 2.505 ms, min = 1.873 us, total = 142.683 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:10:15,282 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 695 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 727 - cumulative published bytes: 579344 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:10:15,283 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 42089 total (4 active) Queueing time: mean = 89.466 us, max = 225.583 ms, min = -0.000 s, total = 3.766 s Execution time: mean = 232.628 us, total = 9.791 s Event stats: event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 25.376 us, total = 353.151 ms, Queueing time: mean = 10.082 us, max = 6.895 ms, min = 1.268 us, total = 140.314 ms GcsInMemoryStore.Put - 5613 total (0 active), Execution time: mean = 45.461 us, total = 255.172 ms, Queueing time: mean = 324.546 us, max = 225.278 ms, min = 1.495 us, total = 1.822 s RayletLoadPulled - 3480 total (1 active), Execution time: mean = 244.751 us, total = 851.733 ms, Queueing time: mean = 73.649 us, max = 1.351 ms, min = 10.621 us, total = 256.299 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3478 total (0 active), Execution time: mean = 75.204 us, total = 261.560 ms, Queueing time: mean = 94.752 us, max = 5.512 ms, min = 11.600 us, total = 329.548 ms NodeManagerService.grpc_client.GetResourceLoad - 3478 total (0 active), Execution time: mean = 1.377 ms, total = 4.788 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1405 total (0 active), Execution time: mean = 503.268 us, total = 707.092 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1405 total (0 active), Execution time: mean = 40.149 us, total = 56.410 ms, Queueing time: mean = 89.681 us, max = 601.063 us, min = 11.893 us, total = 126.002 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1392 total (0 active), Execution time: mean = 35.111 us, total = 48.875 ms, Queueing time: mean = 105.656 us, max = 354.546 us, min = 17.215 us, total = 147.073 ms NodeInfoGcsService.grpc_server.CheckAlive - 1392 total (0 active), Execution time: mean = 524.906 us, total = 730.669 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1161 total (1 active), Execution time: mean = 10.383 us, total = 12.055 ms, Queueing time: mean = 71.448 us, max = 715.636 us, min = 7.259 us, total = 82.951 ms HealthCheck - 1158 total (0 active), Execution time: mean = 7.611 us, total = 8.814 ms, Queueing time: mean = 88.331 us, max = 288.247 us, min = 11.335 us, total = 102.288 ms GcsInMemoryStore.Get - 755 total (0 active), Execution time: mean = 21.358 us, total = 16.125 ms, Queueing time: mean = 12.931 us, max = 137.899 us, min = 2.227 us, total = 9.763 ms InternalKVGcsService.grpc_server.InternalKVGet - 754 total (0 active), Execution time: mean = 629.909 us, total = 474.952 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 754 total (0 active), Execution time: mean = 55.480 us, total = 41.832 ms, Queueing time: mean = 182.578 us, max = 22.982 ms, min = 9.184 us, total = 137.663 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 695 total (0 active), Execution time: mean = 616.461 us, total = 428.440 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 695 total (0 active), Execution time: mean = 90.380 us, total = 62.814 ms, Queueing time: mean = 104.038 us, max = 225.322 us, min = 15.775 us, total = 72.306 ms GCSServer.deadline_timer.debug_state_dump - 348 total (1 active), Execution time: mean = 1.329 ms, total = 462.485 ms, Queueing time: mean = 67.750 us, max = 166.240 us, min = -0.000 s, total = 23.577 ms GCSServer.deadline_timer.debug_state_event_stats_print - 58 total (1 active, 1 running), Execution time: mean = 1.557 ms, total = 90.283 ms, Queueing time: mean = 57.249 us, max = 137.733 us, min = 14.993 us, total = 3.320 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:10:15,283 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 160691 total (1 active) Queueing time: mean = 42.716 us, max = 4.613 ms, min = 1.891 us, total = 6.864 s Execution time: mean = 223.368 us, total = 35.893 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 73038 total (0 active), Execution time: mean = 463.033 us, total = 33.819 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 73038 total (0 active), Execution time: mean = 23.703 us, total = 1.731 s, Queueing time: mean = 91.586 us, max = 4.613 ms, min = 2.005 us, total = 6.689 s event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 24.155 us, total = 336.166 ms, Queueing time: mean = 8.570 us, max = 395.058 us, min = 1.891 us, total = 119.267 ms GcsTaskManager.GcJobSummary - 697 total (1 active), Execution time: mean = 9.754 us, total = 6.799 ms, Queueing time: mean = 79.563 us, max = 3.642 ms, min = 14.860 us, total = 55.456 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:10:15,283 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17168 total (25 active) Queueing time: mean = 16.283 us, max = 600.706 us, min = 2.413 us, total = 279.550 ms Execution time: mean = 4.236 s, total = 72725.832 s Event stats: event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 25.908 us, total = 360.555 ms, Queueing time: mean = 9.417 us, max = 600.706 us, min = 2.413 us, total = 131.054 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 867 total (0 active), Execution time: mean = 37.594 us, total = 32.594 ms, Queueing time: mean = 88.377 us, max = 408.208 us, min = 5.417 us, total = 76.623 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 867 total (24 active), Execution time: mean = 83.881 s, total = 72724.909 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 727 total (0 active), Execution time: mean = 102.988 us, total = 74.872 ms, Queueing time: mean = 94.980 us, max = 214.455 us, min = 11.318 us, total = 69.050 ms InternalPubSubGcsService.grpc_server.GcsPublish - 727 total (0 active), Execution time: mean = 569.065 us, total = 413.711 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 437.940 us, total = 5.255 ms, Queueing time: mean = 69.264 us, max = 114.456 us, min = 34.746 us, total = 831.168 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:10:15,283 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13963 total (0 active) Queueing time: mean = 10.555 us, max = 2.505 ms, min = 563.000 ns, total = 147.386 ms Execution time: mean = 27.383 us, total = 382.342 ms Event stats: event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 27.208 us, total = 378.658 ms, Queueing time: mean = 10.407 us, max = 2.505 ms, min = 1.873 us, total = 144.830 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:11:15,283 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 706 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 739 - cumulative published bytes: 589172 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:11:15,284 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 42793 total (4 active) Queueing time: mean = 88.701 us, max = 225.583 ms, min = -0.000 s, total = 3.796 s Execution time: mean = 232.871 us, total = 9.965 s Event stats: event_loop_lag_probe - 14156 total (0 active), Execution time: mean = 25.438 us, total = 360.105 ms, Queueing time: mean = 10.110 us, max = 6.895 ms, min = 1.268 us, total = 143.123 ms GcsInMemoryStore.Put - 5707 total (0 active), Execution time: mean = 44.789 us, total = 255.612 ms, Queueing time: mean = 319.973 us, max = 225.278 ms, min = 1.495 us, total = 1.826 s RayletLoadPulled - 3539 total (1 active), Execution time: mean = 244.977 us, total = 866.975 ms, Queueing time: mean = 73.708 us, max = 1.351 ms, min = 10.621 us, total = 260.851 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3537 total (0 active), Execution time: mean = 75.263 us, total = 266.204 ms, Queueing time: mean = 94.905 us, max = 5.512 ms, min = 11.600 us, total = 335.678 ms NodeManagerService.grpc_client.GetResourceLoad - 3537 total (0 active), Execution time: mean = 1.379 ms, total = 4.879 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1427 total (0 active), Execution time: mean = 504.656 us, total = 720.145 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1427 total (0 active), Execution time: mean = 40.258 us, total = 57.449 ms, Queueing time: mean = 89.837 us, max = 601.063 us, min = 11.893 us, total = 128.197 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1416 total (0 active), Execution time: mean = 35.110 us, total = 49.716 ms, Queueing time: mean = 105.845 us, max = 354.546 us, min = 17.215 us, total = 149.876 ms NodeInfoGcsService.grpc_server.CheckAlive - 1416 total (0 active), Execution time: mean = 526.370 us, total = 745.340 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1181 total (1 active), Execution time: mean = 10.409 us, total = 12.294 ms, Queueing time: mean = 71.777 us, max = 715.636 us, min = 7.259 us, total = 84.768 ms HealthCheck - 1178 total (0 active), Execution time: mean = 7.684 us, total = 9.051 ms, Queueing time: mean = 88.556 us, max = 288.247 us, min = 11.335 us, total = 104.319 ms GcsInMemoryStore.Get - 766 total (0 active), Execution time: mean = 21.455 us, total = 16.435 ms, Queueing time: mean = 12.984 us, max = 137.899 us, min = 2.227 us, total = 9.946 ms InternalKVGcsService.grpc_server.InternalKVGet - 765 total (0 active), Execution time: mean = 629.644 us, total = 481.677 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 765 total (0 active), Execution time: mean = 55.468 us, total = 42.433 ms, Queueing time: mean = 181.568 us, max = 22.982 ms, min = 9.184 us, total = 138.899 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 706 total (0 active), Execution time: mean = 617.508 us, total = 435.961 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 706 total (0 active), Execution time: mean = 90.465 us, total = 63.868 ms, Queueing time: mean = 104.470 us, max = 225.322 us, min = 15.775 us, total = 73.756 ms GCSServer.deadline_timer.debug_state_dump - 354 total (1 active), Execution time: mean = 1.329 ms, total = 470.628 ms, Queueing time: mean = 68.142 us, max = 166.240 us, min = -0.000 s, total = 24.122 ms GCSServer.deadline_timer.debug_state_event_stats_print - 59 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 91.995 ms, Queueing time: mean = 57.382 us, max = 137.733 us, min = 14.993 us, total = 3.386 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:11:15,284 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 163459 total (1 active) Queueing time: mean = 42.874 us, max = 4.613 ms, min = 1.891 us, total = 7.008 s Execution time: mean = 223.985 us, total = 36.612 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 74296 total (0 active), Execution time: mean = 464.357 us, total = 34.500 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 74296 total (0 active), Execution time: mean = 23.732 us, total = 1.763 s, Queueing time: mean = 91.932 us, max = 4.613 ms, min = 2.005 us, total = 6.830 s event_loop_lag_probe - 14157 total (0 active), Execution time: mean = 24.187 us, total = 342.422 ms, Queueing time: mean = 8.584 us, max = 395.058 us, min = 1.891 us, total = 121.520 ms GcsTaskManager.GcJobSummary - 709 total (1 active), Execution time: mean = 9.771 us, total = 6.927 ms, Queueing time: mean = 79.654 us, max = 3.642 ms, min = 14.860 us, total = 56.474 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:11:15,284 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17456 total (25 active) Queueing time: mean = 16.319 us, max = 600.706 us, min = 2.413 us, total = 284.859 ms Execution time: mean = 4.170 s, total = 72786.006 s Event stats: event_loop_lag_probe - 14157 total (0 active), Execution time: mean = 25.920 us, total = 366.943 ms, Queueing time: mean = 9.419 us, max = 600.706 us, min = 2.413 us, total = 133.347 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 879 total (0 active), Execution time: mean = 37.754 us, total = 33.186 ms, Queueing time: mean = 88.816 us, max = 408.208 us, min = 5.417 us, total = 78.069 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 879 total (24 active), Execution time: mean = 82.804 s, total = 72785.066 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 739 total (0 active), Execution time: mean = 103.403 us, total = 76.415 ms, Queueing time: mean = 95.562 us, max = 236.271 us, min = 11.318 us, total = 70.621 ms InternalPubSubGcsService.grpc_server.GcsPublish - 739 total (0 active), Execution time: mean = 570.910 us, total = 421.902 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 437.940 us, total = 5.255 ms, Queueing time: mean = 69.264 us, max = 114.456 us, min = 34.746 us, total = 831.168 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:11:15,284 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14203 total (0 active) Queueing time: mean = 10.544 us, max = 2.505 ms, min = 563.000 ns, total = 149.762 ms Execution time: mean = 27.379 us, total = 388.866 ms Event stats: event_loop_lag_probe - 14157 total (0 active), Execution time: mean = 27.208 us, total = 385.182 ms, Queueing time: mean = 10.398 us, max = 2.505 ms, min = 1.873 us, total = 147.206 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:12:15,284 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 718 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 751 - cumulative published bytes: 599000 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:12:15,285 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 43512 total (4 active) Queueing time: mean = 87.935 us, max = 225.583 ms, min = -0.000 s, total = 3.826 s Execution time: mean = 233.132 us, total = 10.144 s Event stats: event_loop_lag_probe - 14396 total (0 active), Execution time: mean = 25.491 us, total = 366.972 ms, Queueing time: mean = 10.108 us, max = 6.895 ms, min = 1.268 us, total = 145.510 ms GcsInMemoryStore.Put - 5803 total (0 active), Execution time: mean = 44.135 us, total = 256.114 ms, Queueing time: mean = 315.379 us, max = 225.278 ms, min = 1.495 us, total = 1.830 s RayletLoadPulled - 3599 total (1 active), Execution time: mean = 245.338 us, total = 882.973 ms, Queueing time: mean = 73.971 us, max = 1.351 ms, min = 10.621 us, total = 266.222 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3597 total (0 active), Execution time: mean = 75.384 us, total = 271.156 ms, Queueing time: mean = 95.107 us, max = 5.512 ms, min = 11.600 us, total = 342.099 ms NodeManagerService.grpc_client.GetResourceLoad - 3597 total (0 active), Execution time: mean = 1.382 ms, total = 4.972 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1451 total (0 active), Execution time: mean = 505.459 us, total = 733.420 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1451 total (0 active), Execution time: mean = 40.386 us, total = 58.600 ms, Queueing time: mean = 89.982 us, max = 601.063 us, min = 11.893 us, total = 130.564 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1440 total (0 active), Execution time: mean = 35.205 us, total = 50.695 ms, Queueing time: mean = 106.183 us, max = 354.546 us, min = 17.215 us, total = 152.903 ms NodeInfoGcsService.grpc_server.CheckAlive - 1440 total (0 active), Execution time: mean = 527.885 us, total = 760.155 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1201 total (1 active), Execution time: mean = 10.440 us, total = 12.538 ms, Queueing time: mean = 72.021 us, max = 715.636 us, min = 7.259 us, total = 86.497 ms HealthCheck - 1198 total (0 active), Execution time: mean = 7.686 us, total = 9.207 ms, Queueing time: mean = 88.703 us, max = 288.247 us, min = 11.335 us, total = 106.267 ms GcsInMemoryStore.Get - 778 total (0 active), Execution time: mean = 21.488 us, total = 16.718 ms, Queueing time: mean = 12.970 us, max = 137.899 us, min = 2.227 us, total = 10.090 ms InternalKVGcsService.grpc_server.InternalKVGet - 777 total (0 active), Execution time: mean = 629.797 us, total = 489.352 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 777 total (0 active), Execution time: mean = 55.517 us, total = 43.137 ms, Queueing time: mean = 180.411 us, max = 22.982 ms, min = 9.184 us, total = 140.179 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 718 total (0 active), Execution time: mean = 618.584 us, total = 444.143 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 718 total (0 active), Execution time: mean = 90.438 us, total = 64.934 ms, Queueing time: mean = 104.452 us, max = 225.322 us, min = 15.775 us, total = 74.997 ms GCSServer.deadline_timer.debug_state_dump - 360 total (1 active), Execution time: mean = 1.328 ms, total = 478.017 ms, Queueing time: mean = 68.269 us, max = 166.240 us, min = -0.000 s, total = 24.577 ms GCSServer.deadline_timer.debug_state_event_stats_print - 60 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 92.838 ms, Queueing time: mean = 56.731 us, max = 137.733 us, min = 14.993 us, total = 3.404 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:12:15,285 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 166230 total (1 active) Queueing time: mean = 43.022 us, max = 4.613 ms, min = 1.891 us, total = 7.152 s Execution time: mean = 224.562 us, total = 37.329 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 75556 total (0 active), Execution time: mean = 465.560 us, total = 35.176 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 75556 total (0 active), Execution time: mean = 23.785 us, total = 1.797 s, Queueing time: mean = 92.253 us, max = 4.613 ms, min = 2.005 us, total = 6.970 s event_loop_lag_probe - 14396 total (0 active), Execution time: mean = 24.239 us, total = 348.945 ms, Queueing time: mean = 8.602 us, max = 395.058 us, min = 1.891 us, total = 123.841 ms GcsTaskManager.GcJobSummary - 721 total (1 active), Execution time: mean = 9.783 us, total = 7.054 ms, Queueing time: mean = 79.560 us, max = 3.642 ms, min = 12.167 us, total = 57.363 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:12:15,285 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17789 total (25 active) Queueing time: mean = 16.439 us, max = 600.706 us, min = 2.413 us, total = 292.438 ms Execution time: mean = 4.837 s, total = 86046.189 s Event stats: event_loop_lag_probe - 14397 total (0 active), Execution time: mean = 25.941 us, total = 373.468 ms, Queueing time: mean = 9.423 us, max = 600.706 us, min = 2.413 us, total = 135.668 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 913 total (0 active), Execution time: mean = 37.540 us, total = 34.274 ms, Queueing time: mean = 89.613 us, max = 408.208 us, min = 5.417 us, total = 81.817 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 913 total (24 active), Execution time: mean = 94.245 s, total = 86045.231 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 751 total (0 active), Execution time: mean = 103.611 us, total = 77.812 ms, Queueing time: mean = 95.897 us, max = 236.271 us, min = 11.318 us, total = 72.019 ms InternalPubSubGcsService.grpc_server.GcsPublish - 751 total (0 active), Execution time: mean = 572.309 us, total = 429.804 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 470.619 us, total = 6.118 ms, Queueing time: mean = 72.570 us, max = 114.456 us, min = 34.746 us, total = 943.404 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:12:15,285 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14443 total (0 active) Queueing time: mean = 10.544 us, max = 2.505 ms, min = 563.000 ns, total = 152.284 ms Execution time: mean = 27.405 us, total = 395.812 ms Event stats: event_loop_lag_probe - 14397 total (0 active), Execution time: mean = 27.237 us, total = 392.128 ms, Queueing time: mean = 10.400 us, max = 2.505 ms, min = 1.873 us, total = 149.728 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:13:15,285 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 730 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 763 - cumulative published bytes: 608828 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:13:15,286 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 44231 total (4 active) Queueing time: mean = 87.212 us, max = 225.583 ms, min = -0.000 s, total = 3.857 s Execution time: mean = 233.382 us, total = 10.323 s Event stats: event_loop_lag_probe - 14636 total (0 active), Execution time: mean = 25.542 us, total = 373.830 ms, Queueing time: mean = 10.110 us, max = 6.895 ms, min = 1.268 us, total = 147.976 ms GcsInMemoryStore.Put - 5899 total (0 active), Execution time: mean = 43.508 us, total = 256.655 ms, Queueing time: mean = 311.015 us, max = 225.278 ms, min = 1.495 us, total = 1.835 s RayletLoadPulled - 3659 total (1 active), Execution time: mean = 245.661 us, total = 898.872 ms, Queueing time: mean = 74.155 us, max = 1.351 ms, min = 10.621 us, total = 271.334 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3657 total (0 active), Execution time: mean = 75.448 us, total = 275.912 ms, Queueing time: mean = 95.302 us, max = 5.512 ms, min = 11.600 us, total = 348.521 ms NodeManagerService.grpc_client.GetResourceLoad - 3657 total (0 active), Execution time: mean = 1.385 ms, total = 5.064 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1475 total (0 active), Execution time: mean = 507.071 us, total = 747.929 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1475 total (0 active), Execution time: mean = 40.699 us, total = 60.031 ms, Queueing time: mean = 90.310 us, max = 601.063 us, min = 11.893 us, total = 133.207 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1464 total (0 active), Execution time: mean = 35.250 us, total = 51.605 ms, Queueing time: mean = 106.403 us, max = 354.546 us, min = 17.215 us, total = 155.773 ms NodeInfoGcsService.grpc_server.CheckAlive - 1464 total (0 active), Execution time: mean = 529.207 us, total = 774.760 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1221 total (1 active), Execution time: mean = 10.451 us, total = 12.761 ms, Queueing time: mean = 72.091 us, max = 715.636 us, min = 7.259 us, total = 88.023 ms HealthCheck - 1218 total (0 active), Execution time: mean = 7.709 us, total = 9.389 ms, Queueing time: mean = 88.921 us, max = 288.247 us, min = 11.335 us, total = 108.306 ms GcsInMemoryStore.Get - 790 total (0 active), Execution time: mean = 21.490 us, total = 16.977 ms, Queueing time: mean = 12.980 us, max = 137.899 us, min = 2.227 us, total = 10.254 ms InternalKVGcsService.grpc_server.InternalKVGet - 789 total (0 active), Execution time: mean = 629.707 us, total = 496.839 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 789 total (0 active), Execution time: mean = 55.529 us, total = 43.813 ms, Queueing time: mean = 179.409 us, max = 22.982 ms, min = 9.184 us, total = 141.553 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 730 total (0 active), Execution time: mean = 619.894 us, total = 452.523 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 730 total (0 active), Execution time: mean = 90.490 us, total = 66.058 ms, Queueing time: mean = 104.839 us, max = 225.322 us, min = 15.775 us, total = 76.532 ms GCSServer.deadline_timer.debug_state_dump - 366 total (1 active), Execution time: mean = 1.329 ms, total = 486.463 ms, Queueing time: mean = 68.563 us, max = 166.240 us, min = -0.000 s, total = 25.094 ms GCSServer.deadline_timer.debug_state_event_stats_print - 61 total (1 active, 1 running), Execution time: mean = 1.538 ms, total = 93.816 ms, Queueing time: mean = 56.731 us, max = 137.733 us, min = 14.993 us, total = 3.461 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:13:15,286 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 169002 total (1 active) Queueing time: mean = 43.156 us, max = 4.613 ms, min = 1.891 us, total = 7.293 s Execution time: mean = 225.117 us, total = 38.045 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 76816 total (0 active), Execution time: mean = 466.727 us, total = 35.852 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 76816 total (0 active), Execution time: mean = 23.829 us, total = 1.830 s, Queueing time: mean = 92.548 us, max = 4.613 ms, min = 2.005 us, total = 7.109 s event_loop_lag_probe - 14636 total (0 active), Execution time: mean = 24.281 us, total = 355.377 ms, Queueing time: mean = 8.617 us, max = 395.058 us, min = 1.891 us, total = 126.117 ms GcsTaskManager.GcJobSummary - 733 total (1 active), Execution time: mean = 9.781 us, total = 7.169 ms, Queueing time: mean = 79.397 us, max = 3.642 ms, min = 12.167 us, total = 58.198 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:13:15,286 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18077 total (25 active) Queueing time: mean = 16.462 us, max = 600.706 us, min = 2.413 us, total = 297.578 ms Execution time: mean = 4.763 s, total = 86106.356 s Event stats: event_loop_lag_probe - 14637 total (0 active), Execution time: mean = 25.990 us, total = 380.409 ms, Queueing time: mean = 9.437 us, max = 600.706 us, min = 2.413 us, total = 138.125 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 925 total (0 active), Execution time: mean = 37.742 us, total = 34.911 ms, Queueing time: mean = 89.851 us, max = 408.208 us, min = 5.417 us, total = 83.112 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 925 total (24 active), Execution time: mean = 93.087 s, total = 86105.381 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 763 total (0 active), Execution time: mean = 103.912 us, total = 79.285 ms, Queueing time: mean = 96.207 us, max = 236.271 us, min = 11.318 us, total = 73.406 ms InternalPubSubGcsService.grpc_server.GcsPublish - 763 total (0 active), Execution time: mean = 573.681 us, total = 437.719 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 470.619 us, total = 6.118 ms, Queueing time: mean = 72.570 us, max = 114.456 us, min = 34.746 us, total = 943.404 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:13:15,286 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14683 total (0 active) Queueing time: mean = 10.537 us, max = 2.505 ms, min = 563.000 ns, total = 154.716 ms Execution time: mean = 27.389 us, total = 402.156 ms Event stats: event_loop_lag_probe - 14637 total (0 active), Execution time: mean = 27.224 us, total = 398.471 ms, Queueing time: mean = 10.396 us, max = 2.505 ms, min = 1.873 us, total = 152.160 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:14:15,287 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 742 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 776 - cumulative published bytes: 618690 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:14:15,288 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 44950 total (4 active) Queueing time: mean = 86.532 us, max = 225.583 ms, min = -0.000 s, total = 3.890 s Execution time: mean = 233.514 us, total = 10.496 s Event stats: event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 25.597 us, total = 380.787 ms, Queueing time: mean = 10.116 us, max = 6.895 ms, min = 1.268 us, total = 150.478 ms GcsInMemoryStore.Put - 5995 total (0 active), Execution time: mean = 42.899 us, total = 257.177 ms, Queueing time: mean = 306.757 us, max = 225.278 ms, min = 1.495 us, total = 1.839 s RayletLoadPulled - 3719 total (1 active), Execution time: mean = 246.175 us, total = 915.524 ms, Queueing time: mean = 74.359 us, max = 1.351 ms, min = 10.621 us, total = 276.541 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3717 total (0 active), Execution time: mean = 75.531 us, total = 280.748 ms, Queueing time: mean = 95.497 us, max = 5.512 ms, min = 11.600 us, total = 354.962 ms NodeManagerService.grpc_client.GetResourceLoad - 3717 total (0 active), Execution time: mean = 1.386 ms, total = 5.152 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1499 total (0 active), Execution time: mean = 507.641 us, total = 760.953 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1499 total (0 active), Execution time: mean = 40.690 us, total = 60.994 ms, Queueing time: mean = 90.422 us, max = 601.063 us, min = 11.893 us, total = 135.542 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1488 total (0 active), Execution time: mean = 35.266 us, total = 52.475 ms, Queueing time: mean = 107.595 us, max = 1.496 ms, min = 17.215 us, total = 160.102 ms NodeInfoGcsService.grpc_server.CheckAlive - 1488 total (0 active), Execution time: mean = 531.052 us, total = 790.205 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1241 total (1 active), Execution time: mean = 10.464 us, total = 12.986 ms, Queueing time: mean = 72.437 us, max = 715.636 us, min = 7.259 us, total = 89.894 ms HealthCheck - 1238 total (0 active), Execution time: mean = 7.717 us, total = 9.553 ms, Queueing time: mean = 89.050 us, max = 288.247 us, min = 11.335 us, total = 110.243 ms GcsInMemoryStore.Get - 802 total (0 active), Execution time: mean = 21.450 us, total = 17.203 ms, Queueing time: mean = 12.967 us, max = 137.899 us, min = 2.227 us, total = 10.400 ms InternalKVGcsService.grpc_server.InternalKVGet - 801 total (0 active), Execution time: mean = 628.507 us, total = 503.434 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 801 total (0 active), Execution time: mean = 55.564 us, total = 44.507 ms, Queueing time: mean = 178.082 us, max = 22.982 ms, min = 9.184 us, total = 142.644 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 742 total (0 active), Execution time: mean = 620.505 us, total = 460.414 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 742 total (0 active), Execution time: mean = 90.519 us, total = 67.165 ms, Queueing time: mean = 105.116 us, max = 225.322 us, min = 15.775 us, total = 77.996 ms GCSServer.deadline_timer.debug_state_dump - 372 total (1 active), Execution time: mean = 1.330 ms, total = 494.621 ms, Queueing time: mean = 68.641 us, max = 166.240 us, min = -0.000 s, total = 25.534 ms GCSServer.deadline_timer.debug_state_event_stats_print - 62 total (1 active, 1 running), Execution time: mean = 1.536 ms, total = 95.255 ms, Queueing time: mean = 56.835 us, max = 137.733 us, min = 14.993 us, total = 3.524 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:14:15,288 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 171774 total (1 active) Queueing time: mean = 43.249 us, max = 4.613 ms, min = 1.891 us, total = 7.429 s Execution time: mean = 225.429 us, total = 38.723 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 78076 total (0 active), Execution time: mean = 467.375 us, total = 36.491 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 78076 total (0 active), Execution time: mean = 23.860 us, total = 1.863 s, Queueing time: mean = 92.749 us, max = 4.613 ms, min = 2.005 us, total = 7.241 s event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 24.325 us, total = 361.859 ms, Queueing time: mean = 8.635 us, max = 395.058 us, min = 1.891 us, total = 128.455 ms GcsTaskManager.GcJobSummary - 745 total (1 active), Execution time: mean = 9.790 us, total = 7.293 ms, Queueing time: mean = 79.411 us, max = 3.642 ms, min = 12.167 us, total = 59.161 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:14:15,288 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18368 total (25 active) Queueing time: mean = 16.458 us, max = 600.706 us, min = 2.413 us, total = 302.294 ms Execution time: mean = 4.691 s, total = 86166.498 s Event stats: event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 26.030 us, total = 387.215 ms, Queueing time: mean = 9.449 us, max = 600.706 us, min = 2.413 us, total = 140.569 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 938 total (0 active), Execution time: mean = 37.806 us, total = 35.462 ms, Queueing time: mean = 89.796 us, max = 408.208 us, min = 5.417 us, total = 84.228 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 938 total (24 active), Execution time: mean = 91.861 s, total = 86165.508 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 776 total (0 active), Execution time: mean = 103.807 us, total = 80.554 ms, Queueing time: mean = 96.086 us, max = 236.271 us, min = 11.318 us, total = 74.563 ms InternalPubSubGcsService.grpc_server.GcsPublish - 776 total (0 active), Execution time: mean = 573.712 us, total = 445.201 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 470.619 us, total = 6.118 ms, Queueing time: mean = 72.570 us, max = 114.456 us, min = 34.746 us, total = 943.404 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:14:15,288 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14922 total (0 active) Queueing time: mean = 10.533 us, max = 2.505 ms, min = 563.000 ns, total = 157.179 ms Execution time: mean = 27.399 us, total = 408.846 ms Event stats: event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 27.236 us, total = 405.162 ms, Queueing time: mean = 10.394 us, max = 2.505 ms, min = 1.873 us, total = 154.623 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:15:15,289 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 754 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 788 - cumulative published bytes: 628518 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:15:15,290 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 45669 total (4 active) Queueing time: mean = 85.855 us, max = 225.583 ms, min = -0.000 s, total = 3.921 s Execution time: mean = 233.799 us, total = 10.677 s Event stats: event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 25.642 us, total = 387.601 ms, Queueing time: mean = 10.118 us, max = 6.895 ms, min = 1.268 us, total = 152.939 ms GcsInMemoryStore.Put - 6091 total (0 active), Execution time: mean = 42.313 us, total = 257.726 ms, Queueing time: mean = 302.756 us, max = 225.278 ms, min = 1.495 us, total = 1.844 s RayletLoadPulled - 3779 total (1 active), Execution time: mean = 246.410 us, total = 931.184 ms, Queueing time: mean = 74.512 us, max = 1.351 ms, min = 10.621 us, total = 281.580 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3777 total (0 active), Execution time: mean = 75.655 us, total = 285.750 ms, Queueing time: mean = 95.533 us, max = 5.512 ms, min = 11.600 us, total = 360.828 ms NodeManagerService.grpc_client.GetResourceLoad - 3777 total (0 active), Execution time: mean = 1.389 ms, total = 5.245 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1523 total (0 active), Execution time: mean = 508.815 us, total = 774.925 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1523 total (0 active), Execution time: mean = 40.865 us, total = 62.237 ms, Queueing time: mean = 90.627 us, max = 601.063 us, min = 11.893 us, total = 138.024 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1512 total (0 active), Execution time: mean = 35.488 us, total = 53.659 ms, Queueing time: mean = 107.981 us, max = 1.496 ms, min = 17.215 us, total = 163.268 ms NodeInfoGcsService.grpc_server.CheckAlive - 1512 total (0 active), Execution time: mean = 532.393 us, total = 804.978 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1261 total (1 active), Execution time: mean = 10.490 us, total = 13.228 ms, Queueing time: mean = 72.669 us, max = 715.636 us, min = 7.259 us, total = 91.636 ms HealthCheck - 1258 total (0 active), Execution time: mean = 7.723 us, total = 9.715 ms, Queueing time: mean = 89.242 us, max = 288.247 us, min = 11.335 us, total = 112.267 ms GcsInMemoryStore.Get - 814 total (0 active), Execution time: mean = 21.428 us, total = 17.443 ms, Queueing time: mean = 12.973 us, max = 137.899 us, min = 2.227 us, total = 10.560 ms InternalKVGcsService.grpc_server.InternalKVGet - 813 total (0 active), Execution time: mean = 628.390 us, total = 510.881 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 813 total (0 active), Execution time: mean = 55.774 us, total = 45.344 ms, Queueing time: mean = 177.043 us, max = 22.982 ms, min = 9.184 us, total = 143.936 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 754 total (0 active), Execution time: mean = 621.838 us, total = 468.866 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 754 total (0 active), Execution time: mean = 90.539 us, total = 68.266 ms, Queueing time: mean = 105.334 us, max = 225.322 us, min = 15.775 us, total = 79.422 ms GCSServer.deadline_timer.debug_state_dump - 378 total (1 active), Execution time: mean = 1.331 ms, total = 502.955 ms, Queueing time: mean = 68.781 us, max = 166.240 us, min = -0.000 s, total = 25.999 ms GCSServer.deadline_timer.debug_state_event_stats_print - 63 total (1 active, 1 running), Execution time: mean = 1.540 ms, total = 97.002 ms, Queueing time: mean = 56.975 us, max = 137.733 us, min = 14.993 us, total = 3.589 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:15:15,290 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 174546 total (1 active) Queueing time: mean = 43.398 us, max = 4.613 ms, min = 1.891 us, total = 7.575 s Execution time: mean = 226.014 us, total = 39.450 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 79336 total (0 active), Execution time: mean = 468.591 us, total = 37.176 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 79336 total (0 active), Execution time: mean = 23.916 us, total = 1.897 s, Queueing time: mean = 93.069 us, max = 4.613 ms, min = 2.005 us, total = 7.384 s event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 24.399 us, total = 368.819 ms, Queueing time: mean = 8.671 us, max = 395.058 us, min = 1.891 us, total = 131.075 ms GcsTaskManager.GcJobSummary - 757 total (1 active), Execution time: mean = 9.834 us, total = 7.445 ms, Queueing time: mean = 79.471 us, max = 3.642 ms, min = 12.167 us, total = 60.160 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:15:15,290 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18656 total (25 active) Queueing time: mean = 16.484 us, max = 600.706 us, min = 2.413 us, total = 307.518 ms Execution time: mean = 4.622 s, total = 86226.685 s Event stats: event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 26.074 us, total = 394.139 ms, Queueing time: mean = 9.466 us, max = 600.706 us, min = 2.413 us, total = 143.095 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 950 total (0 active), Execution time: mean = 37.877 us, total = 35.984 ms, Queueing time: mean = 89.997 us, max = 408.208 us, min = 5.417 us, total = 85.497 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 950 total (24 active), Execution time: mean = 90.764 s, total = 86225.678 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 788 total (0 active), Execution time: mean = 103.750 us, total = 81.755 ms, Queueing time: mean = 96.435 us, max = 236.271 us, min = 11.318 us, total = 75.991 ms InternalPubSubGcsService.grpc_server.GcsPublish - 788 total (0 active), Execution time: mean = 574.760 us, total = 452.911 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 470.619 us, total = 6.118 ms, Queueing time: mean = 72.570 us, max = 114.456 us, min = 34.746 us, total = 943.404 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:15:15,290 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15162 total (0 active) Queueing time: mean = 10.527 us, max = 2.505 ms, min = 563.000 ns, total = 159.614 ms Execution time: mean = 27.391 us, total = 415.310 ms Event stats: event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 27.231 us, total = 411.625 ms, Queueing time: mean = 10.390 us, max = 2.505 ms, min = 1.873 us, total = 157.058 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:16:15,290 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 766 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 800 - cumulative published bytes: 638346 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:16:15,291 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 46386 total (4 active) Queueing time: mean = 85.165 us, max = 225.583 ms, min = -0.000 s, total = 3.950 s Execution time: mean = 234.006 us, total = 10.855 s Event stats: event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 25.689 us, total = 394.481 ms, Queueing time: mean = 10.123 us, max = 6.895 ms, min = 1.268 us, total = 155.448 ms GcsInMemoryStore.Put - 6187 total (0 active), Execution time: mean = 41.736 us, total = 258.222 ms, Queueing time: mean = 298.721 us, max = 225.278 ms, min = 1.495 us, total = 1.848 s RayletLoadPulled - 3839 total (1 active), Execution time: mean = 246.737 us, total = 947.223 ms, Queueing time: mean = 74.625 us, max = 1.351 ms, min = 10.621 us, total = 286.484 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3837 total (0 active), Execution time: mean = 75.722 us, total = 290.547 ms, Queueing time: mean = 95.618 us, max = 5.512 ms, min = 11.600 us, total = 366.885 ms NodeManagerService.grpc_client.GetResourceLoad - 3837 total (0 active), Execution time: mean = 1.391 ms, total = 5.337 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1547 total (0 active), Execution time: mean = 509.966 us, total = 788.917 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1547 total (0 active), Execution time: mean = 40.892 us, total = 63.260 ms, Queueing time: mean = 90.828 us, max = 601.063 us, min = 11.893 us, total = 140.510 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1535 total (0 active), Execution time: mean = 35.609 us, total = 54.660 ms, Queueing time: mean = 108.126 us, max = 1.496 ms, min = 17.215 us, total = 165.973 ms NodeInfoGcsService.grpc_server.CheckAlive - 1535 total (0 active), Execution time: mean = 533.155 us, total = 818.392 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1281 total (1 active), Execution time: mean = 10.499 us, total = 13.449 ms, Queueing time: mean = 72.840 us, max = 715.636 us, min = 7.259 us, total = 93.308 ms HealthCheck - 1278 total (0 active), Execution time: mean = 7.737 us, total = 9.888 ms, Queueing time: mean = 89.456 us, max = 288.247 us, min = 11.335 us, total = 114.325 ms GcsInMemoryStore.Get - 826 total (0 active), Execution time: mean = 21.407 us, total = 17.682 ms, Queueing time: mean = 12.964 us, max = 137.899 us, min = 2.227 us, total = 10.708 ms InternalKVGcsService.grpc_server.InternalKVGet - 825 total (0 active), Execution time: mean = 627.862 us, total = 517.986 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 825 total (0 active), Execution time: mean = 55.865 us, total = 46.088 ms, Queueing time: mean = 175.939 us, max = 22.982 ms, min = 9.184 us, total = 145.150 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 766 total (0 active), Execution time: mean = 622.994 us, total = 477.213 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 766 total (0 active), Execution time: mean = 90.729 us, total = 69.498 ms, Queueing time: mean = 105.485 us, max = 225.322 us, min = 15.775 us, total = 80.801 ms GCSServer.deadline_timer.debug_state_dump - 384 total (1 active), Execution time: mean = 1.329 ms, total = 510.495 ms, Queueing time: mean = 68.405 us, max = 166.240 us, min = -0.000 s, total = 26.267 ms GCSServer.deadline_timer.debug_state_event_stats_print - 64 total (1 active, 1 running), Execution time: mean = 1.543 ms, total = 98.744 ms, Queueing time: mean = 57.086 us, max = 137.733 us, min = 14.993 us, total = 3.654 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:16:15,291 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 177318 total (1 active) Queueing time: mean = 43.545 us, max = 4.613 ms, min = 1.891 us, total = 7.721 s Execution time: mean = 226.513 us, total = 40.165 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 80596 total (0 active), Execution time: mean = 469.639 us, total = 37.851 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 80596 total (0 active), Execution time: mean = 23.958 us, total = 1.931 s, Queueing time: mean = 93.389 us, max = 4.613 ms, min = 2.005 us, total = 7.527 s event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 24.441 us, total = 375.310 ms, Queueing time: mean = 8.688 us, max = 395.058 us, min = 1.891 us, total = 133.414 ms GcsTaskManager.GcJobSummary - 769 total (1 active), Execution time: mean = 9.850 us, total = 7.575 ms, Queueing time: mean = 79.452 us, max = 3.642 ms, min = 12.167 us, total = 61.099 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:16:15,291 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18944 total (25 active) Queueing time: mean = 16.492 us, max = 600.706 us, min = 2.413 us, total = 312.425 ms Execution time: mean = 4.555 s, total = 86286.887 s Event stats: event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 26.138 us, total = 401.370 ms, Queueing time: mean = 9.479 us, max = 600.706 us, min = 2.413 us, total = 145.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 962 total (0 active), Execution time: mean = 38.089 us, total = 36.641 ms, Queueing time: mean = 89.879 us, max = 408.208 us, min = 5.417 us, total = 86.464 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 962 total (24 active), Execution time: mean = 89.694 s, total = 86285.863 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 800 total (0 active), Execution time: mean = 104.081 us, total = 83.265 ms, Queueing time: mean = 96.835 us, max = 236.271 us, min = 11.318 us, total = 77.468 ms InternalPubSubGcsService.grpc_server.GcsPublish - 800 total (0 active), Execution time: mean = 575.541 us, total = 460.433 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 470.619 us, total = 6.118 ms, Queueing time: mean = 72.570 us, max = 114.456 us, min = 34.746 us, total = 943.404 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:16:15,291 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15402 total (0 active) Queueing time: mean = 10.520 us, max = 2.505 ms, min = 563.000 ns, total = 162.024 ms Execution time: mean = 27.391 us, total = 421.877 ms Event stats: event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 27.233 us, total = 418.193 ms, Queueing time: mean = 10.385 us, max = 2.505 ms, min = 1.873 us, total = 159.468 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:17:15,292 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 778 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 813 - cumulative published bytes: 648208 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:17:15,293 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 47105 total (4 active) Queueing time: mean = 84.036 us, max = 225.583 ms, min = -0.023 s, total = 3.959 s Execution time: mean = 234.630 us, total = 11.052 s Event stats: event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 25.712 us, total = 400.998 ms, Queueing time: mean = 10.118 us, max = 6.895 ms, min = 1.268 us, total = 157.796 ms GcsInMemoryStore.Put - 6283 total (0 active), Execution time: mean = 41.182 us, total = 258.749 ms, Queueing time: mean = 294.932 us, max = 225.278 ms, min = 1.495 us, total = 1.853 s RayletLoadPulled - 3899 total (1 active), Execution time: mean = 246.944 us, total = 962.836 ms, Queueing time: mean = 74.867 us, max = 1.351 ms, min = 10.621 us, total = 291.907 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3897 total (0 active), Execution time: mean = 75.811 us, total = 295.436 ms, Queueing time: mean = 95.746 us, max = 5.512 ms, min = 11.600 us, total = 373.123 ms NodeManagerService.grpc_client.GetResourceLoad - 3897 total (0 active), Execution time: mean = 1.392 ms, total = 5.426 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1571 total (0 active), Execution time: mean = 512.872 us, total = 805.722 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1571 total (0 active), Execution time: mean = 40.812 us, total = 64.116 ms, Queueing time: mean = 90.835 us, max = 601.063 us, min = 11.893 us, total = 142.702 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1559 total (0 active), Execution time: mean = 35.622 us, total = 55.535 ms, Queueing time: mean = 108.299 us, max = 1.496 ms, min = 17.215 us, total = 168.838 ms NodeInfoGcsService.grpc_server.CheckAlive - 1559 total (0 active), Execution time: mean = 534.152 us, total = 832.742 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1301 total (1 active), Execution time: mean = 10.484 us, total = 13.639 ms, Queueing time: mean = 72.963 us, max = 715.636 us, min = 7.259 us, total = 94.925 ms HealthCheck - 1298 total (0 active), Execution time: mean = 7.743 us, total = 10.051 ms, Queueing time: mean = 89.693 us, max = 288.247 us, min = 11.335 us, total = 116.421 ms GcsInMemoryStore.Get - 838 total (0 active), Execution time: mean = 21.448 us, total = 17.973 ms, Queueing time: mean = 12.944 us, max = 137.899 us, min = 2.227 us, total = 10.847 ms InternalKVGcsService.grpc_server.InternalKVGet - 837 total (0 active), Execution time: mean = 626.670 us, total = 524.523 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 837 total (0 active), Execution time: mean = 55.769 us, total = 46.679 ms, Queueing time: mean = 174.807 us, max = 22.982 ms, min = 9.184 us, total = 146.313 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 778 total (0 active), Execution time: mean = 623.167 us, total = 484.824 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 778 total (0 active), Execution time: mean = 90.707 us, total = 70.570 ms, Queueing time: mean = 105.524 us, max = 235.695 us, min = 15.775 us, total = 82.098 ms GCSServer.deadline_timer.debug_state_dump - 390 total (1 active), Execution time: mean = 1.388 ms, total = 541.386 ms, Queueing time: mean = 10.310 us, max = 166.240 us, min = -0.023 s, total = 4.021 ms GCSServer.deadline_timer.debug_state_event_stats_print - 65 total (1 active, 1 running), Execution time: mean = 1.537 ms, total = 99.903 ms, Queueing time: mean = 57.037 us, max = 137.733 us, min = 14.993 us, total = 3.707 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:17:15,293 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 180090 total (1 active) Queueing time: mean = 43.650 us, max = 4.613 ms, min = 1.891 us, total = 7.861 s Execution time: mean = 226.902 us, total = 40.863 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 81856 total (0 active), Execution time: mean = 470.488 us, total = 38.512 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 81856 total (0 active), Execution time: mean = 23.952 us, total = 1.961 s, Queueing time: mean = 93.615 us, max = 4.613 ms, min = 2.005 us, total = 7.663 s event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 24.510 us, total = 382.255 ms, Queueing time: mean = 8.707 us, max = 395.058 us, min = 1.891 us, total = 135.797 ms GcsTaskManager.GcJobSummary - 781 total (1 active), Execution time: mean = 9.872 us, total = 7.710 ms, Queueing time: mean = 79.467 us, max = 3.642 ms, min = 12.167 us, total = 62.064 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:17:15,293 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19239 total (25 active) Queueing time: mean = 16.517 us, max = 600.706 us, min = 2.413 us, total = 317.765 ms Execution time: mean = 4.519 s, total = 86946.992 s Event stats: event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 26.162 us, total = 408.017 ms, Queueing time: mean = 9.483 us, max = 600.706 us, min = 2.413 us, total = 147.892 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 976 total (0 active), Execution time: mean = 38.125 us, total = 37.210 ms, Queueing time: mean = 90.135 us, max = 408.208 us, min = 5.417 us, total = 87.972 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 976 total (24 active), Execution time: mean = 89.084 s, total = 86945.951 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 813 total (0 active), Execution time: mean = 104.105 us, total = 84.637 ms, Queueing time: mean = 97.037 us, max = 236.271 us, min = 11.318 us, total = 78.891 ms InternalPubSubGcsService.grpc_server.GcsPublish - 813 total (0 active), Execution time: mean = 576.272 us, total = 468.509 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 440.332 us, total = 6.165 ms, Queueing time: mean = 72.763 us, max = 114.456 us, min = 34.746 us, total = 1.019 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:17:15,293 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15642 total (0 active) Queueing time: mean = 10.509 us, max = 2.505 ms, min = 563.000 ns, total = 164.385 ms Execution time: mean = 27.391 us, total = 428.447 ms Event stats: event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 27.235 us, total = 424.763 ms, Queueing time: mean = 10.376 us, max = 2.505 ms, min = 1.873 us, total = 161.828 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:18:15,293 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 790 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 827 - cumulative published bytes: 658104 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:18:15,294 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 47824 total (4 active) Queueing time: mean = 83.333 us, max = 225.583 ms, min = -0.023 s, total = 3.985 s Execution time: mean = 234.416 us, total = 11.211 s Event stats: event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 25.728 us, total = 407.431 ms, Queueing time: mean = 10.110 us, max = 6.895 ms, min = 1.268 us, total = 160.095 ms GcsInMemoryStore.Put - 6379 total (0 active), Execution time: mean = 40.645 us, total = 259.273 ms, Queueing time: mean = 291.266 us, max = 225.278 ms, min = 1.495 us, total = 1.858 s RayletLoadPulled - 3959 total (1 active), Execution time: mean = 246.942 us, total = 977.643 ms, Queueing time: mean = 74.873 us, max = 1.351 ms, min = 10.621 us, total = 296.421 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3957 total (0 active), Execution time: mean = 75.868 us, total = 300.211 ms, Queueing time: mean = 95.659 us, max = 5.512 ms, min = 11.600 us, total = 378.522 ms NodeManagerService.grpc_client.GetResourceLoad - 3957 total (0 active), Execution time: mean = 1.392 ms, total = 5.507 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1595 total (0 active), Execution time: mean = 512.842 us, total = 817.982 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1595 total (0 active), Execution time: mean = 40.792 us, total = 65.063 ms, Queueing time: mean = 90.603 us, max = 601.063 us, min = 11.490 us, total = 144.512 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1583 total (0 active), Execution time: mean = 35.589 us, total = 56.338 ms, Queueing time: mean = 108.127 us, max = 1.496 ms, min = 17.215 us, total = 171.165 ms NodeInfoGcsService.grpc_server.CheckAlive - 1583 total (0 active), Execution time: mean = 533.257 us, total = 844.145 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1321 total (1 active), Execution time: mean = 10.472 us, total = 13.833 ms, Queueing time: mean = 72.914 us, max = 715.636 us, min = 7.259 us, total = 96.320 ms HealthCheck - 1318 total (0 active), Execution time: mean = 7.736 us, total = 10.196 ms, Queueing time: mean = 89.564 us, max = 288.247 us, min = 11.335 us, total = 118.045 ms GcsInMemoryStore.Get - 850 total (0 active), Execution time: mean = 21.448 us, total = 18.231 ms, Queueing time: mean = 12.919 us, max = 137.899 us, min = 2.227 us, total = 10.981 ms InternalKVGcsService.grpc_server.InternalKVGet - 849 total (0 active), Execution time: mean = 625.477 us, total = 531.030 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 849 total (0 active), Execution time: mean = 55.877 us, total = 47.440 ms, Queueing time: mean = 173.464 us, max = 22.982 ms, min = 9.184 us, total = 147.271 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 790 total (0 active), Execution time: mean = 622.615 us, total = 491.866 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 790 total (0 active), Execution time: mean = 90.599 us, total = 71.573 ms, Queueing time: mean = 105.171 us, max = 235.695 us, min = 15.511 us, total = 83.085 ms GCSServer.deadline_timer.debug_state_dump - 396 total (1 active), Execution time: mean = 1.387 ms, total = 549.351 ms, Queueing time: mean = 11.011 us, max = 166.240 us, min = -0.023 s, total = 4.360 ms GCSServer.deadline_timer.debug_state_event_stats_print - 66 total (1 active, 1 running), Execution time: mean = 1.539 ms, total = 101.573 ms, Queueing time: mean = 57.138 us, max = 137.733 us, min = 14.993 us, total = 3.771 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:18:15,294 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 182860 total (1 active) Queueing time: mean = 43.594 us, max = 4.613 ms, min = 1.891 us, total = 7.972 s Execution time: mean = 226.799 us, total = 41.472 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 83115 total (0 active), Execution time: mean = 470.278 us, total = 39.087 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 83115 total (0 active), Execution time: mean = 23.928 us, total = 1.989 s, Queueing time: mean = 93.493 us, max = 4.613 ms, min = 2.005 us, total = 7.771 s event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 24.545 us, total = 388.697 ms, Queueing time: mean = 8.717 us, max = 395.058 us, min = 1.891 us, total = 138.043 ms GcsTaskManager.GcJobSummary - 793 total (1 active), Execution time: mean = 9.890 us, total = 7.843 ms, Queueing time: mean = 79.207 us, max = 3.642 ms, min = 12.167 us, total = 62.811 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:18:15,295 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19535 total (25 active) Queueing time: mean = 16.489 us, max = 600.706 us, min = 2.413 us, total = 322.113 ms Execution time: mean = 4.454 s, total = 87007.189 s Event stats: event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 26.140 us, total = 413.955 ms, Queueing time: mean = 9.475 us, max = 600.706 us, min = 2.413 us, total = 150.050 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 990 total (0 active), Execution time: mean = 38.122 us, total = 37.740 ms, Queueing time: mean = 90.020 us, max = 408.208 us, min = 5.417 us, total = 89.120 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 990 total (24 active), Execution time: mean = 87.885 s, total = 87006.134 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 827 total (0 active), Execution time: mean = 103.958 us, total = 85.974 ms, Queueing time: mean = 96.654 us, max = 236.271 us, min = 11.318 us, total = 79.933 ms InternalPubSubGcsService.grpc_server.GcsPublish - 827 total (0 active), Execution time: mean = 574.362 us, total = 474.997 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 440.332 us, total = 6.165 ms, Queueing time: mean = 72.763 us, max = 114.456 us, min = 34.746 us, total = 1.019 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:18:15,295 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15882 total (0 active) Queueing time: mean = 10.492 us, max = 2.505 ms, min = 563.000 ns, total = 166.634 ms Execution time: mean = 27.356 us, total = 434.467 ms Event stats: event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 27.203 us, total = 430.783 ms, Queueing time: mean = 10.361 us, max = 2.505 ms, min = 1.873 us, total = 164.077 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:19:15,295 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 802 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 840 - cumulative published bytes: 667966 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:19:15,296 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 48543 total (4 active) Queueing time: mean = 82.681 us, max = 225.583 ms, min = -0.023 s, total = 4.014 s Execution time: mean = 234.306 us, total = 11.374 s Event stats: event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 25.742 us, total = 413.828 ms, Queueing time: mean = 10.102 us, max = 6.895 ms, min = 1.268 us, total = 162.405 ms GcsInMemoryStore.Put - 6475 total (0 active), Execution time: mean = 40.116 us, total = 259.754 ms, Queueing time: mean = 287.569 us, max = 225.278 ms, min = 1.495 us, total = 1.862 s RayletLoadPulled - 4019 total (1 active), Execution time: mean = 247.057 us, total = 992.923 ms, Queueing time: mean = 74.897 us, max = 1.351 ms, min = 10.621 us, total = 301.010 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4017 total (0 active), Execution time: mean = 75.902 us, total = 304.897 ms, Queueing time: mean = 95.635 us, max = 5.512 ms, min = 11.600 us, total = 384.166 ms NodeManagerService.grpc_client.GetResourceLoad - 4017 total (0 active), Execution time: mean = 1.391 ms, total = 5.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1619 total (0 active), Execution time: mean = 513.329 us, total = 831.080 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1619 total (0 active), Execution time: mean = 40.861 us, total = 66.154 ms, Queueing time: mean = 90.749 us, max = 601.063 us, min = 10.186 us, total = 146.923 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1607 total (0 active), Execution time: mean = 35.567 us, total = 57.156 ms, Queueing time: mean = 108.215 us, max = 1.496 ms, min = 14.438 us, total = 173.902 ms NodeInfoGcsService.grpc_server.CheckAlive - 1607 total (0 active), Execution time: mean = 533.291 us, total = 856.998 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1341 total (1 active), Execution time: mean = 10.451 us, total = 14.015 ms, Queueing time: mean = 72.918 us, max = 715.636 us, min = 7.259 us, total = 97.783 ms HealthCheck - 1338 total (0 active), Execution time: mean = 7.738 us, total = 10.353 ms, Queueing time: mean = 89.608 us, max = 288.247 us, min = 11.335 us, total = 119.896 ms GcsInMemoryStore.Get - 862 total (0 active), Execution time: mean = 21.419 us, total = 18.463 ms, Queueing time: mean = 12.933 us, max = 137.899 us, min = 2.227 us, total = 11.148 ms InternalKVGcsService.grpc_server.InternalKVGet - 861 total (0 active), Execution time: mean = 624.384 us, total = 537.595 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 861 total (0 active), Execution time: mean = 55.914 us, total = 48.142 ms, Queueing time: mean = 172.463 us, max = 22.982 ms, min = 9.184 us, total = 148.490 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 802 total (0 active), Execution time: mean = 622.517 us, total = 499.259 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 802 total (0 active), Execution time: mean = 90.469 us, total = 72.556 ms, Queueing time: mean = 105.165 us, max = 235.695 us, min = 15.511 us, total = 84.342 ms GCSServer.deadline_timer.debug_state_dump - 402 total (1 active), Execution time: mean = 1.389 ms, total = 558.280 ms, Queueing time: mean = 12.230 us, max = 166.240 us, min = -0.023 s, total = 4.916 ms GCSServer.deadline_timer.debug_state_event_stats_print - 67 total (1 active, 1 running), Execution time: mean = 1.544 ms, total = 103.433 ms, Queueing time: mean = 57.252 us, max = 137.733 us, min = 14.993 us, total = 3.836 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:19:15,296 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 185632 total (1 active) Queueing time: mean = 43.598 us, max = 4.613 ms, min = 1.891 us, total = 8.093 s Execution time: mean = 226.842 us, total = 42.109 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 84375 total (0 active), Execution time: mean = 470.370 us, total = 39.687 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 84375 total (0 active), Execution time: mean = 23.923 us, total = 2.018 s, Queueing time: mean = 93.503 us, max = 4.613 ms, min = 2.005 us, total = 7.889 s event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 24.587 us, total = 395.266 ms, Queueing time: mean = 8.728 us, max = 395.058 us, min = 1.891 us, total = 140.316 ms GcsTaskManager.GcJobSummary - 805 total (1 active), Execution time: mean = 9.883 us, total = 7.956 ms, Queueing time: mean = 78.991 us, max = 3.642 ms, min = 12.167 us, total = 63.588 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:19:15,296 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19827 total (25 active) Queueing time: mean = 16.468 us, max = 600.706 us, min = 2.413 us, total = 326.513 ms Execution time: mean = 4.391 s, total = 87067.391 s Event stats: event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 26.147 us, total = 420.346 ms, Queueing time: mean = 9.475 us, max = 600.706 us, min = 2.413 us, total = 152.312 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1003 total (0 active), Execution time: mean = 38.249 us, total = 38.363 ms, Queueing time: mean = 89.922 us, max = 408.208 us, min = 5.417 us, total = 90.192 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1003 total (24 active), Execution time: mean = 86.806 s, total = 87066.321 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 840 total (0 active), Execution time: mean = 103.804 us, total = 87.195 ms, Queueing time: mean = 96.427 us, max = 236.271 us, min = 11.318 us, total = 80.998 ms InternalPubSubGcsService.grpc_server.GcsPublish - 840 total (0 active), Execution time: mean = 573.632 us, total = 481.851 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 440.332 us, total = 6.165 ms, Queueing time: mean = 72.763 us, max = 114.456 us, min = 34.746 us, total = 1.019 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:19:15,297 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16122 total (0 active) Queueing time: mean = 10.478 us, max = 2.505 ms, min = 563.000 ns, total = 168.927 ms Execution time: mean = 27.324 us, total = 440.516 ms Event stats: event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 27.173 us, total = 436.832 ms, Queueing time: mean = 10.349 us, max = 2.505 ms, min = 1.873 us, total = 166.371 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:20:15,297 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 814 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 852 - cumulative published bytes: 677794 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:20:15,298 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 49261 total (4 active) Queueing time: mean = 82.001 us, max = 225.583 ms, min = -0.023 s, total = 4.039 s Execution time: mean = 233.975 us, total = 11.526 s Event stats: event_loop_lag_probe - 16315 total (0 active), Execution time: mean = 25.752 us, total = 420.144 ms, Queueing time: mean = 10.090 us, max = 6.895 ms, min = 1.268 us, total = 164.620 ms GcsInMemoryStore.Put - 6571 total (0 active), Execution time: mean = 39.602 us, total = 260.223 ms, Queueing time: mean = 283.991 us, max = 225.278 ms, min = 1.495 us, total = 1.866 s RayletLoadPulled - 4079 total (1 active), Execution time: mean = 246.877 us, total = 1.007 s, Queueing time: mean = 74.815 us, max = 1.351 ms, min = 10.621 us, total = 305.170 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4077 total (0 active), Execution time: mean = 75.830 us, total = 309.159 ms, Queueing time: mean = 95.433 us, max = 5.512 ms, min = 11.600 us, total = 389.082 ms NodeManagerService.grpc_client.GetResourceLoad - 4077 total (0 active), Execution time: mean = 1.389 ms, total = 5.665 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1643 total (0 active), Execution time: mean = 513.223 us, total = 843.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1643 total (0 active), Execution time: mean = 40.770 us, total = 66.986 ms, Queueing time: mean = 90.636 us, max = 601.063 us, min = 10.186 us, total = 148.915 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1631 total (0 active), Execution time: mean = 35.527 us, total = 57.944 ms, Queueing time: mean = 108.039 us, max = 1.496 ms, min = 14.438 us, total = 176.211 ms NodeInfoGcsService.grpc_server.CheckAlive - 1631 total (0 active), Execution time: mean = 532.674 us, total = 868.792 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1361 total (1 active), Execution time: mean = 10.440 us, total = 14.209 ms, Queueing time: mean = 72.827 us, max = 715.636 us, min = 7.259 us, total = 99.117 ms HealthCheck - 1358 total (0 active), Execution time: mean = 7.730 us, total = 10.498 ms, Queueing time: mean = 89.662 us, max = 288.247 us, min = 11.335 us, total = 121.761 ms GcsInMemoryStore.Get - 874 total (0 active), Execution time: mean = 21.418 us, total = 18.719 ms, Queueing time: mean = 12.959 us, max = 137.899 us, min = 2.227 us, total = 11.326 ms InternalKVGcsService.grpc_server.InternalKVGet - 873 total (0 active), Execution time: mean = 623.254 us, total = 544.101 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 873 total (0 active), Execution time: mean = 55.886 us, total = 48.788 ms, Queueing time: mean = 171.311 us, max = 22.982 ms, min = 9.184 us, total = 149.555 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 814 total (0 active), Execution time: mean = 622.766 us, total = 506.932 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 814 total (0 active), Execution time: mean = 90.482 us, total = 73.653 ms, Queueing time: mean = 105.339 us, max = 235.695 us, min = 15.511 us, total = 85.746 ms GCSServer.deadline_timer.debug_state_dump - 408 total (1 active), Execution time: mean = 1.385 ms, total = 565.052 ms, Queueing time: mean = 12.710 us, max = 166.240 us, min = -0.023 s, total = 5.186 ms GCSServer.deadline_timer.debug_state_event_stats_print - 68 total (1 active, 1 running), Execution time: mean = 1.544 ms, total = 104.990 ms, Queueing time: mean = 57.012 us, max = 137.733 us, min = 14.993 us, total = 3.877 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:20:15,298 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 188404 total (1 active) Queueing time: mean = 43.548 us, max = 4.613 ms, min = 1.891 us, total = 8.205 s Execution time: mean = 226.642 us, total = 42.700 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 85635 total (0 active), Execution time: mean = 469.984 us, total = 40.247 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 85635 total (0 active), Execution time: mean = 23.864 us, total = 2.044 s, Queueing time: mean = 93.394 us, max = 4.613 ms, min = 2.005 us, total = 7.998 s event_loop_lag_probe - 16316 total (0 active), Execution time: mean = 24.609 us, total = 401.526 ms, Queueing time: mean = 8.735 us, max = 395.058 us, min = 1.891 us, total = 142.524 ms GcsTaskManager.GcJobSummary - 817 total (1 active), Execution time: mean = 9.861 us, total = 8.057 ms, Queueing time: mean = 78.711 us, max = 3.642 ms, min = 12.167 us, total = 64.307 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:20:15,298 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20115 total (25 active) Queueing time: mean = 16.463 us, max = 600.706 us, min = 2.413 us, total = 331.157 ms Execution time: mean = 4.331 s, total = 87127.500 s Event stats: event_loop_lag_probe - 16316 total (0 active), Execution time: mean = 26.220 us, total = 427.812 ms, Queueing time: mean = 9.474 us, max = 600.706 us, min = 2.413 us, total = 154.579 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1015 total (0 active), Execution time: mean = 38.209 us, total = 38.782 ms, Queueing time: mean = 89.984 us, max = 408.208 us, min = 5.417 us, total = 91.334 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1015 total (24 active), Execution time: mean = 85.839 s, total = 87126.413 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 852 total (0 active), Execution time: mean = 104.165 us, total = 88.749 ms, Queueing time: mean = 96.519 us, max = 236.271 us, min = 11.318 us, total = 82.234 ms InternalPubSubGcsService.grpc_server.GcsPublish - 852 total (0 active), Execution time: mean = 574.319 us, total = 489.320 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 440.332 us, total = 6.165 ms, Queueing time: mean = 72.763 us, max = 114.456 us, min = 34.746 us, total = 1.019 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:20:15,298 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16362 total (0 active) Queueing time: mean = 10.465 us, max = 2.505 ms, min = 563.000 ns, total = 171.233 ms Execution time: mean = 27.291 us, total = 446.531 ms Event stats: event_loop_lag_probe - 16316 total (0 active), Execution time: mean = 27.142 us, total = 442.847 ms, Queueing time: mean = 10.338 us, max = 2.505 ms, min = 1.873 us, total = 168.677 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:21:15,299 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 826 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 864 - cumulative published bytes: 687622 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:21:15,300 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 49980 total (4 active) Queueing time: mean = 81.335 us, max = 225.583 ms, min = -0.023 s, total = 4.065 s Execution time: mean = 233.304 us, total = 11.661 s Event stats: event_loop_lag_probe - 16555 total (0 active), Execution time: mean = 25.742 us, total = 426.162 ms, Queueing time: mean = 10.079 us, max = 6.895 ms, min = 1.268 us, total = 166.857 ms GcsInMemoryStore.Put - 6667 total (0 active), Execution time: mean = 39.094 us, total = 260.641 ms, Queueing time: mean = 280.516 us, max = 225.278 ms, min = 1.495 us, total = 1.870 s RayletLoadPulled - 4139 total (1 active), Execution time: mean = 246.509 us, total = 1.020 s, Queueing time: mean = 74.795 us, max = 1.351 ms, min = 10.621 us, total = 309.578 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4137 total (0 active), Execution time: mean = 75.747 us, total = 313.364 ms, Queueing time: mean = 95.267 us, max = 5.512 ms, min = 11.600 us, total = 394.120 ms NodeManagerService.grpc_client.GetResourceLoad - 4137 total (0 active), Execution time: mean = 1.385 ms, total = 5.732 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1667 total (0 active), Execution time: mean = 510.761 us, total = 851.438 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1667 total (0 active), Execution time: mean = 40.695 us, total = 67.838 ms, Queueing time: mean = 90.315 us, max = 601.063 us, min = 10.186 us, total = 150.555 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1655 total (0 active), Execution time: mean = 35.546 us, total = 58.829 ms, Queueing time: mean = 107.812 us, max = 1.496 ms, min = 14.438 us, total = 178.428 ms NodeInfoGcsService.grpc_server.CheckAlive - 1655 total (0 active), Execution time: mean = 531.194 us, total = 879.127 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1381 total (1 active), Execution time: mean = 10.427 us, total = 14.400 ms, Queueing time: mean = 72.870 us, max = 715.636 us, min = 7.259 us, total = 100.633 ms HealthCheck - 1378 total (0 active), Execution time: mean = 7.714 us, total = 10.630 ms, Queueing time: mean = 89.596 us, max = 288.247 us, min = 11.335 us, total = 123.463 ms GcsInMemoryStore.Get - 886 total (0 active), Execution time: mean = 21.359 us, total = 18.924 ms, Queueing time: mean = 12.936 us, max = 137.899 us, min = 2.227 us, total = 11.461 ms InternalKVGcsService.grpc_server.InternalKVGet - 885 total (0 active), Execution time: mean = 620.426 us, total = 549.077 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 885 total (0 active), Execution time: mean = 55.825 us, total = 49.405 ms, Queueing time: mean = 170.241 us, max = 22.982 ms, min = 9.184 us, total = 150.663 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 826 total (0 active), Execution time: mean = 620.898 us, total = 512.862 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 826 total (0 active), Execution time: mean = 90.390 us, total = 74.662 ms, Queueing time: mean = 105.191 us, max = 235.695 us, min = 15.511 us, total = 86.888 ms GCSServer.deadline_timer.debug_state_dump - 414 total (1 active), Execution time: mean = 1.386 ms, total = 573.807 ms, Queueing time: mean = 13.460 us, max = 166.240 us, min = -0.023 s, total = 5.572 ms GCSServer.deadline_timer.debug_state_event_stats_print - 69 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 106.761 ms, Queueing time: mean = 57.144 us, max = 137.733 us, min = 14.993 us, total = 3.943 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:21:15,300 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 191176 total (2 active) Queueing time: mean = 43.241 us, max = 4.613 ms, min = 1.891 us, total = 8.267 s Execution time: mean = 225.425 us, total = 43.096 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 86895 total (1 active), Execution time: mean = 467.404 us, total = 40.615 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 86895 total (0 active), Execution time: mean = 23.766 us, total = 2.065 s, Queueing time: mean = 92.723 us, max = 4.613 ms, min = 2.005 us, total = 8.057 s event_loop_lag_probe - 16556 total (0 active), Execution time: mean = 24.609 us, total = 407.434 ms, Queueing time: mean = 8.726 us, max = 395.058 us, min = 1.891 us, total = 144.468 ms GcsTaskManager.GcJobSummary - 829 total (1 active), Execution time: mean = 9.861 us, total = 8.175 ms, Queueing time: mean = 78.326 us, max = 3.642 ms, min = 12.167 us, total = 64.933 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:21:15,300 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20403 total (25 active) Queueing time: mean = 16.379 us, max = 600.706 us, min = 2.413 us, total = 334.180 ms Execution time: mean = 4.273 s, total = 87187.698 s Event stats: event_loop_lag_probe - 16556 total (0 active), Execution time: mean = 26.181 us, total = 433.456 ms, Queueing time: mean = 9.464 us, max = 600.706 us, min = 2.413 us, total = 156.693 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1027 total (0 active), Execution time: mean = 38.107 us, total = 39.136 ms, Queueing time: mean = 89.363 us, max = 408.208 us, min = 5.417 us, total = 91.775 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1027 total (24 active), Execution time: mean = 84.894 s, total = 87186.599 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 864 total (0 active), Execution time: mean = 103.959 us, total = 89.821 ms, Queueing time: mean = 95.720 us, max = 236.271 us, min = 11.318 us, total = 82.702 ms InternalPubSubGcsService.grpc_server.GcsPublish - 864 total (0 active), Execution time: mean = 571.778 us, total = 494.016 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 440.332 us, total = 6.165 ms, Queueing time: mean = 72.763 us, max = 114.456 us, min = 34.746 us, total = 1.019 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:21:15,300 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16602 total (0 active) Queueing time: mean = 10.446 us, max = 2.505 ms, min = 563.000 ns, total = 173.419 ms Execution time: mean = 27.246 us, total = 452.346 ms Event stats: event_loop_lag_probe - 16556 total (0 active), Execution time: mean = 27.100 us, total = 448.661 ms, Queueing time: mean = 10.320 us, max = 2.505 ms, min = 1.873 us, total = 170.863 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:22:15,300 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 838 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 876 - cumulative published bytes: 697450 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:22:15,301 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 50699 total (4 active) Queueing time: mean = 80.783 us, max = 225.583 ms, min = -0.023 s, total = 4.096 s Execution time: mean = 233.480 us, total = 11.837 s Event stats: event_loop_lag_probe - 16795 total (0 active), Execution time: mean = 25.780 us, total = 432.979 ms, Queueing time: mean = 10.085 us, max = 6.895 ms, min = 1.268 us, total = 169.379 ms GcsInMemoryStore.Put - 6763 total (0 active), Execution time: mean = 38.611 us, total = 261.128 ms, Queueing time: mean = 277.135 us, max = 225.278 ms, min = 1.495 us, total = 1.874 s RayletLoadPulled - 4199 total (1 active), Execution time: mean = 246.679 us, total = 1.036 s, Queueing time: mean = 75.024 us, max = 1.351 ms, min = 10.621 us, total = 315.026 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4197 total (0 active), Execution time: mean = 75.762 us, total = 317.973 ms, Queueing time: mean = 95.406 us, max = 5.512 ms, min = 11.600 us, total = 400.419 ms NodeManagerService.grpc_client.GetResourceLoad - 4197 total (0 active), Execution time: mean = 1.388 ms, total = 5.826 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1691 total (0 active), Execution time: mean = 511.323 us, total = 864.647 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1691 total (0 active), Execution time: mean = 40.772 us, total = 68.945 ms, Queueing time: mean = 90.514 us, max = 601.063 us, min = 10.186 us, total = 153.060 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1679 total (0 active), Execution time: mean = 35.537 us, total = 59.666 ms, Queueing time: mean = 107.813 us, max = 1.496 ms, min = 14.438 us, total = 181.019 ms NodeInfoGcsService.grpc_server.CheckAlive - 1679 total (0 active), Execution time: mean = 531.499 us, total = 892.386 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1401 total (1 active), Execution time: mean = 10.441 us, total = 14.627 ms, Queueing time: mean = 73.027 us, max = 715.636 us, min = 7.259 us, total = 102.311 ms HealthCheck - 1398 total (0 active), Execution time: mean = 7.714 us, total = 10.785 ms, Queueing time: mean = 89.732 us, max = 288.247 us, min = 11.335 us, total = 125.445 ms GcsInMemoryStore.Get - 898 total (0 active), Execution time: mean = 21.339 us, total = 19.162 ms, Queueing time: mean = 12.922 us, max = 137.899 us, min = 2.227 us, total = 11.604 ms InternalKVGcsService.grpc_server.InternalKVGet - 897 total (0 active), Execution time: mean = 620.112 us, total = 556.240 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 897 total (0 active), Execution time: mean = 55.757 us, total = 50.014 ms, Queueing time: mean = 169.556 us, max = 22.982 ms, min = 9.184 us, total = 152.092 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 838 total (0 active), Execution time: mean = 621.396 us, total = 520.730 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 838 total (0 active), Execution time: mean = 90.691 us, total = 75.999 ms, Queueing time: mean = 105.337 us, max = 235.695 us, min = 15.511 us, total = 88.272 ms GCSServer.deadline_timer.debug_state_dump - 420 total (1 active), Execution time: mean = 1.384 ms, total = 581.304 ms, Queueing time: mean = 14.220 us, max = 166.240 us, min = -0.023 s, total = 5.972 ms GCSServer.deadline_timer.debug_state_event_stats_print - 70 total (1 active, 1 running), Execution time: mean = 1.550 ms, total = 108.504 ms, Queueing time: mean = 57.119 us, max = 137.733 us, min = 14.993 us, total = 3.998 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:22:15,302 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 193911 total (1 active) Queueing time: mean = 43.344 us, max = 4.613 ms, min = 1.891 us, total = 8.405 s Execution time: mean = 225.886 us, total = 43.802 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 88137 total (0 active), Execution time: mean = 468.392 us, total = 41.283 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 88137 total (0 active), Execution time: mean = 23.790 us, total = 2.097 s, Queueing time: mean = 92.951 us, max = 4.613 ms, min = 2.005 us, total = 8.192 s event_loop_lag_probe - 16795 total (0 active), Execution time: mean = 24.648 us, total = 413.962 ms, Queueing time: mean = 8.730 us, max = 395.058 us, min = 1.891 us, total = 146.621 ms GcsTaskManager.GcJobSummary - 841 total (1 active), Execution time: mean = 9.860 us, total = 8.292 ms, Queueing time: mean = 78.346 us, max = 3.642 ms, min = 12.167 us, total = 65.889 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:22:15,302 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20736 total (25 active) Queueing time: mean = 16.461 us, max = 600.706 us, min = 2.413 us, total = 341.336 ms Execution time: mean = 4.844 s, total = 100447.884 s Event stats: event_loop_lag_probe - 16796 total (0 active), Execution time: mean = 26.179 us, total = 439.696 ms, Queueing time: mean = 9.463 us, max = 600.706 us, min = 2.413 us, total = 158.938 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1061 total (0 active), Execution time: mean = 37.870 us, total = 40.180 ms, Queueing time: mean = 89.777 us, max = 408.208 us, min = 5.417 us, total = 95.253 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1061 total (24 active), Execution time: mean = 94.672 s, total = 100446.767 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 876 total (0 active), Execution time: mean = 104.220 us, total = 91.296 ms, Queueing time: mean = 95.947 us, max = 236.271 us, min = 11.318 us, total = 84.050 ms InternalPubSubGcsService.grpc_server.GcsPublish - 876 total (0 active), Execution time: mean = 572.607 us, total = 501.604 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 509.726 us, total = 7.646 ms, Queueing time: mean = 73.593 us, max = 114.456 us, min = 34.746 us, total = 1.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:22:15,302 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16842 total (0 active) Queueing time: mean = 10.437 us, max = 2.505 ms, min = 563.000 ns, total = 175.775 ms Execution time: mean = 27.241 us, total = 458.789 ms Event stats: event_loop_lag_probe - 16796 total (0 active), Execution time: mean = 27.096 us, total = 455.105 ms, Queueing time: mean = 10.313 us, max = 2.505 ms, min = 1.873 us, total = 173.219 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:23:15,302 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 850 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 888 - cumulative published bytes: 707278 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:23:15,303 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 51418 total (4 active) Queueing time: mean = 80.183 us, max = 225.583 ms, min = -0.023 s, total = 4.123 s Execution time: mean = 233.608 us, total = 12.012 s Event stats: event_loop_lag_probe - 17035 total (0 active), Execution time: mean = 25.791 us, total = 439.352 ms, Queueing time: mean = 10.083 us, max = 6.895 ms, min = 1.268 us, total = 171.767 ms GcsInMemoryStore.Put - 6859 total (0 active), Execution time: mean = 38.141 us, total = 261.610 ms, Queueing time: mean = 273.960 us, max = 225.278 ms, min = 1.495 us, total = 1.879 s RayletLoadPulled - 4259 total (1 active), Execution time: mean = 246.667 us, total = 1.051 s, Queueing time: mean = 75.013 us, max = 1.351 ms, min = 10.621 us, total = 319.482 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4257 total (0 active), Execution time: mean = 75.817 us, total = 322.753 ms, Queueing time: mean = 95.416 us, max = 5.512 ms, min = 11.600 us, total = 406.186 ms NodeManagerService.grpc_client.GetResourceLoad - 4257 total (0 active), Execution time: mean = 1.390 ms, total = 5.918 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1715 total (0 active), Execution time: mean = 511.966 us, total = 878.022 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1715 total (0 active), Execution time: mean = 40.788 us, total = 69.951 ms, Queueing time: mean = 90.437 us, max = 601.063 us, min = 10.186 us, total = 155.100 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1703 total (0 active), Execution time: mean = 35.525 us, total = 60.500 ms, Queueing time: mean = 107.520 us, max = 1.496 ms, min = 14.438 us, total = 183.106 ms NodeInfoGcsService.grpc_server.CheckAlive - 1703 total (0 active), Execution time: mean = 531.986 us, total = 905.973 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1421 total (1 active), Execution time: mean = 10.428 us, total = 14.819 ms, Queueing time: mean = 73.069 us, max = 715.636 us, min = 7.259 us, total = 103.831 ms HealthCheck - 1418 total (0 active), Execution time: mean = 7.691 us, total = 10.906 ms, Queueing time: mean = 89.301 us, max = 288.247 us, min = 11.335 us, total = 126.628 ms GcsInMemoryStore.Get - 910 total (0 active), Execution time: mean = 21.380 us, total = 19.456 ms, Queueing time: mean = 12.933 us, max = 137.899 us, min = 2.227 us, total = 11.769 ms InternalKVGcsService.grpc_server.InternalKVGet - 909 total (0 active), Execution time: mean = 619.649 us, total = 563.261 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 909 total (0 active), Execution time: mean = 55.929 us, total = 50.840 ms, Queueing time: mean = 168.543 us, max = 22.982 ms, min = 9.184 us, total = 153.205 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 850 total (0 active), Execution time: mean = 622.489 us, total = 529.115 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 850 total (0 active), Execution time: mean = 91.228 us, total = 77.544 ms, Queueing time: mean = 105.406 us, max = 235.695 us, min = 15.511 us, total = 89.595 ms GCSServer.deadline_timer.debug_state_dump - 426 total (1 active), Execution time: mean = 1.382 ms, total = 588.700 ms, Queueing time: mean = 14.653 us, max = 166.240 us, min = -0.023 s, total = 6.242 ms GCSServer.deadline_timer.debug_state_event_stats_print - 71 total (1 active, 1 running), Execution time: mean = 1.552 ms, total = 110.191 ms, Queueing time: mean = 57.230 us, max = 137.733 us, min = 14.993 us, total = 4.063 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:23:15,303 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 196683 total (1 active) Queueing time: mean = 43.427 us, max = 4.613 ms, min = 1.891 us, total = 8.541 s Execution time: mean = 226.188 us, total = 44.487 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 89397 total (0 active), Execution time: mean = 469.023 us, total = 41.929 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 89397 total (0 active), Execution time: mean = 23.820 us, total = 2.129 s, Queueing time: mean = 93.134 us, max = 4.613 ms, min = 2.005 us, total = 8.326 s event_loop_lag_probe - 17035 total (0 active), Execution time: mean = 24.667 us, total = 420.204 ms, Queueing time: mean = 8.733 us, max = 395.058 us, min = 1.891 us, total = 148.764 ms GcsTaskManager.GcJobSummary - 853 total (1 active), Execution time: mean = 9.876 us, total = 8.424 ms, Queueing time: mean = 78.264 us, max = 3.642 ms, min = 12.167 us, total = 66.760 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:23:15,304 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21024 total (25 active) Queueing time: mean = 16.469 us, max = 600.706 us, min = 2.413 us, total = 346.248 ms Execution time: mean = 4.781 s, total = 100507.972 s Event stats: event_loop_lag_probe - 17036 total (0 active), Execution time: mean = 26.177 us, total = 445.953 ms, Queueing time: mean = 9.460 us, max = 600.706 us, min = 2.413 us, total = 161.161 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1073 total (0 active), Execution time: mean = 37.938 us, total = 40.707 ms, Queueing time: mean = 89.999 us, max = 408.208 us, min = 5.417 us, total = 96.569 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1073 total (24 active), Execution time: mean = 93.669 s, total = 100506.840 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 888 total (0 active), Execution time: mean = 104.391 us, total = 92.699 ms, Queueing time: mean = 96.197 us, max = 236.271 us, min = 11.318 us, total = 85.423 ms InternalPubSubGcsService.grpc_server.GcsPublish - 888 total (0 active), Execution time: mean = 573.543 us, total = 509.306 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 509.726 us, total = 7.646 ms, Queueing time: mean = 73.593 us, max = 114.456 us, min = 34.746 us, total = 1.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:23:15,304 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17082 total (0 active) Queueing time: mean = 10.428 us, max = 2.505 ms, min = 563.000 ns, total = 178.139 ms Execution time: mean = 27.230 us, total = 465.147 ms Event stats: event_loop_lag_probe - 17036 total (0 active), Execution time: mean = 27.087 us, total = 461.462 ms, Queueing time: mean = 10.307 us, max = 2.505 ms, min = 1.873 us, total = 175.583 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:24:15,304 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 862 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 900 - cumulative published bytes: 717106 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:24:15,305 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 52137 total (4 active) Queueing time: mean = 79.661 us, max = 225.583 ms, min = -0.023 s, total = 4.153 s Execution time: mean = 233.625 us, total = 12.181 s Event stats: event_loop_lag_probe - 17275 total (0 active), Execution time: mean = 25.821 us, total = 446.051 ms, Queueing time: mean = 10.089 us, max = 6.895 ms, min = 1.268 us, total = 174.294 ms GcsInMemoryStore.Put - 6955 total (0 active), Execution time: mean = 37.691 us, total = 262.142 ms, Queueing time: mean = 270.786 us, max = 225.278 ms, min = 1.495 us, total = 1.883 s RayletLoadPulled - 4319 total (1 active), Execution time: mean = 246.739 us, total = 1.066 s, Queueing time: mean = 75.049 us, max = 1.351 ms, min = 10.621 us, total = 324.136 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4317 total (0 active), Execution time: mean = 75.846 us, total = 327.425 ms, Queueing time: mean = 95.544 us, max = 5.512 ms, min = 11.600 us, total = 412.462 ms NodeManagerService.grpc_client.GetResourceLoad - 4317 total (0 active), Execution time: mean = 1.391 ms, total = 6.004 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1739 total (0 active), Execution time: mean = 512.490 us, total = 891.221 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1739 total (0 active), Execution time: mean = 40.852 us, total = 71.041 ms, Queueing time: mean = 90.777 us, max = 601.063 us, min = 10.186 us, total = 157.861 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1727 total (0 active), Execution time: mean = 35.508 us, total = 61.322 ms, Queueing time: mean = 107.500 us, max = 1.496 ms, min = 14.438 us, total = 185.652 ms NodeInfoGcsService.grpc_server.CheckAlive - 1727 total (0 active), Execution time: mean = 532.083 us, total = 918.907 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1441 total (1 active), Execution time: mean = 10.434 us, total = 15.036 ms, Queueing time: mean = 73.117 us, max = 715.636 us, min = 7.259 us, total = 105.362 ms HealthCheck - 1438 total (0 active), Execution time: mean = 7.689 us, total = 11.056 ms, Queueing time: mean = 89.615 us, max = 288.247 us, min = 11.335 us, total = 128.866 ms GcsInMemoryStore.Get - 922 total (0 active), Execution time: mean = 21.398 us, total = 19.729 ms, Queueing time: mean = 12.945 us, max = 137.899 us, min = 2.227 us, total = 11.935 ms InternalKVGcsService.grpc_server.InternalKVGet - 921 total (0 active), Execution time: mean = 619.293 us, total = 570.368 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 921 total (0 active), Execution time: mean = 55.975 us, total = 51.553 ms, Queueing time: mean = 167.816 us, max = 22.982 ms, min = 9.184 us, total = 154.559 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 862 total (0 active), Execution time: mean = 623.160 us, total = 537.164 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 862 total (0 active), Execution time: mean = 91.347 us, total = 78.741 ms, Queueing time: mean = 105.815 us, max = 235.695 us, min = 15.511 us, total = 91.213 ms GCSServer.deadline_timer.debug_state_dump - 432 total (1 active), Execution time: mean = 1.382 ms, total = 596.928 ms, Queueing time: mean = 15.687 us, max = 166.240 us, min = -0.023 s, total = 6.777 ms GCSServer.deadline_timer.debug_state_event_stats_print - 72 total (1 active, 1 running), Execution time: mean = 1.556 ms, total = 112.012 ms, Queueing time: mean = 56.886 us, max = 137.733 us, min = 14.993 us, total = 4.096 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:24:15,305 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 199455 total (1 active) Queueing time: mean = 43.397 us, max = 4.613 ms, min = 1.891 us, total = 8.656 s Execution time: mean = 226.104 us, total = 45.098 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 90657 total (0 active), Execution time: mean = 468.823 us, total = 42.502 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 90657 total (0 active), Execution time: mean = 23.814 us, total = 2.159 s, Queueing time: mean = 93.066 us, max = 4.613 ms, min = 2.005 us, total = 8.437 s event_loop_lag_probe - 17275 total (0 active), Execution time: mean = 24.778 us, total = 428.047 ms, Queueing time: mean = 8.749 us, max = 395.058 us, min = 1.891 us, total = 151.138 ms GcsTaskManager.GcJobSummary - 865 total (1 active), Execution time: mean = 9.861 us, total = 8.530 ms, Queueing time: mean = 77.948 us, max = 3.642 ms, min = 12.167 us, total = 67.425 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:24:15,305 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21312 total (25 active) Queueing time: mean = 16.459 us, max = 600.706 us, min = 2.413 us, total = 350.777 ms Execution time: mean = 4.719 s, total = 100568.236 s Event stats: event_loop_lag_probe - 17276 total (0 active), Execution time: mean = 26.197 us, total = 452.582 ms, Queueing time: mean = 9.463 us, max = 600.706 us, min = 2.413 us, total = 163.477 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1085 total (0 active), Execution time: mean = 37.996 us, total = 41.225 ms, Queueing time: mean = 90.003 us, max = 408.208 us, min = 5.417 us, total = 97.653 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1085 total (24 active), Execution time: mean = 92.689 s, total = 100567.088 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 900 total (0 active), Execution time: mean = 104.474 us, total = 94.026 ms, Queueing time: mean = 96.169 us, max = 236.271 us, min = 11.318 us, total = 86.552 ms InternalPubSubGcsService.grpc_server.GcsPublish - 900 total (0 active), Execution time: mean = 573.770 us, total = 516.393 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 509.726 us, total = 7.646 ms, Queueing time: mean = 73.593 us, max = 114.456 us, min = 34.746 us, total = 1.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:24:15,305 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17322 total (0 active) Queueing time: mean = 10.419 us, max = 2.505 ms, min = 563.000 ns, total = 180.478 ms Execution time: mean = 27.220 us, total = 471.508 ms Event stats: event_loop_lag_probe - 17276 total (0 active), Execution time: mean = 27.079 us, total = 467.823 ms, Queueing time: mean = 10.299 us, max = 2.505 ms, min = 1.873 us, total = 177.922 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:25:15,306 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 874 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 912 - cumulative published bytes: 726934 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:25:15,307 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 52856 total (4 active) Queueing time: mean = 79.118 us, max = 225.583 ms, min = -0.023 s, total = 4.182 s Execution time: mean = 233.426 us, total = 12.338 s Event stats: event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 25.838 us, total = 452.554 ms, Queueing time: mean = 10.092 us, max = 6.895 ms, min = 1.268 us, total = 176.762 ms GcsInMemoryStore.Put - 7051 total (0 active), Execution time: mean = 37.246 us, total = 262.624 ms, Queueing time: mean = 267.697 us, max = 225.278 ms, min = 1.495 us, total = 1.888 s RayletLoadPulled - 4379 total (1 active), Execution time: mean = 246.735 us, total = 1.080 s, Queueing time: mean = 75.071 us, max = 1.351 ms, min = 10.621 us, total = 328.736 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4377 total (0 active), Execution time: mean = 76.022 us, total = 332.750 ms, Queueing time: mean = 95.546 us, max = 5.512 ms, min = 11.600 us, total = 418.206 ms NodeManagerService.grpc_client.GetResourceLoad - 4377 total (0 active), Execution time: mean = 1.389 ms, total = 6.082 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1763 total (0 active), Execution time: mean = 512.342 us, total = 903.258 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1763 total (0 active), Execution time: mean = 40.991 us, total = 72.268 ms, Queueing time: mean = 90.830 us, max = 601.063 us, min = 10.186 us, total = 160.133 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1751 total (0 active), Execution time: mean = 35.502 us, total = 62.163 ms, Queueing time: mean = 107.456 us, max = 1.496 ms, min = 14.438 us, total = 188.155 ms NodeInfoGcsService.grpc_server.CheckAlive - 1751 total (0 active), Execution time: mean = 531.620 us, total = 930.866 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1461 total (1 active), Execution time: mean = 10.427 us, total = 15.234 ms, Queueing time: mean = 73.145 us, max = 715.636 us, min = 7.259 us, total = 106.864 ms HealthCheck - 1458 total (0 active), Execution time: mean = 7.680 us, total = 11.197 ms, Queueing time: mean = 89.712 us, max = 288.247 us, min = 11.335 us, total = 130.800 ms GcsInMemoryStore.Get - 934 total (0 active), Execution time: mean = 21.357 us, total = 19.947 ms, Queueing time: mean = 12.935 us, max = 137.899 us, min = 2.227 us, total = 12.081 ms InternalKVGcsService.grpc_server.InternalKVGet - 933 total (0 active), Execution time: mean = 617.861 us, total = 576.464 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 933 total (0 active), Execution time: mean = 55.992 us, total = 52.240 ms, Queueing time: mean = 166.889 us, max = 22.982 ms, min = 9.184 us, total = 155.707 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 874 total (0 active), Execution time: mean = 622.733 us, total = 544.268 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 874 total (0 active), Execution time: mean = 91.446 us, total = 79.924 ms, Queueing time: mean = 105.984 us, max = 235.695 us, min = 15.511 us, total = 92.630 ms GCSServer.deadline_timer.debug_state_dump - 438 total (1 active), Execution time: mean = 1.383 ms, total = 605.874 ms, Queueing time: mean = 16.684 us, max = 166.240 us, min = -0.023 s, total = 7.308 ms GCSServer.deadline_timer.debug_state_event_stats_print - 73 total (1 active, 1 running), Execution time: mean = 1.557 ms, total = 113.629 ms, Queueing time: mean = 56.919 us, max = 137.733 us, min = 14.993 us, total = 4.155 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:25:15,307 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 202227 total (1 active) Queueing time: mean = 43.278 us, max = 4.613 ms, min = 1.891 us, total = 8.752 s Execution time: mean = 225.766 us, total = 45.656 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 91917 total (0 active), Execution time: mean = 468.107 us, total = 43.027 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 91917 total (0 active), Execution time: mean = 23.782 us, total = 2.186 s, Queueing time: mean = 92.806 us, max = 4.613 ms, min = 2.005 us, total = 8.530 s event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 24.799 us, total = 434.363 ms, Queueing time: mean = 8.755 us, max = 395.058 us, min = 1.891 us, total = 153.338 ms GcsTaskManager.GcJobSummary - 877 total (1 active), Execution time: mean = 9.884 us, total = 8.668 ms, Queueing time: mean = 77.786 us, max = 3.642 ms, min = 12.167 us, total = 68.218 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:25:15,307 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21599 total (25 active) Queueing time: mean = 16.437 us, max = 600.706 us, min = 2.413 us, total = 355.030 ms Execution time: mean = 4.659 s, total = 100628.313 s Event stats: event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 26.199 us, total = 458.884 ms, Queueing time: mean = 9.463 us, max = 600.706 us, min = 2.413 us, total = 165.740 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1097 total (0 active), Execution time: mean = 38.070 us, total = 41.763 ms, Queueing time: mean = 89.975 us, max = 408.208 us, min = 5.417 us, total = 98.703 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1097 total (24 active), Execution time: mean = 91.729 s, total = 100627.150 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 912 total (0 active), Execution time: mean = 104.707 us, total = 95.493 ms, Queueing time: mean = 95.934 us, max = 236.271 us, min = 11.318 us, total = 87.492 ms InternalPubSubGcsService.grpc_server.GcsPublish - 912 total (0 active), Execution time: mean = 573.447 us, total = 522.984 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 509.726 us, total = 7.646 ms, Queueing time: mean = 73.593 us, max = 114.456 us, min = 34.746 us, total = 1.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:25:15,307 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17561 total (0 active) Queueing time: mean = 10.413 us, max = 2.505 ms, min = 563.000 ns, total = 182.854 ms Execution time: mean = 27.216 us, total = 477.944 ms Event stats: event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 27.077 us, total = 474.260 ms, Queueing time: mean = 10.294 us, max = 2.505 ms, min = 1.873 us, total = 180.298 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:26:15,307 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 886 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 925 - cumulative published bytes: 736796 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:26:15,308 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 53575 total (4 active) Queueing time: mean = 78.633 us, max = 225.583 ms, min = -0.023 s, total = 4.213 s Execution time: mean = 233.262 us, total = 12.497 s Event stats: event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 25.865 us, total = 459.242 ms, Queueing time: mean = 10.096 us, max = 6.895 ms, min = 1.268 us, total = 179.258 ms GcsInMemoryStore.Put - 7147 total (0 active), Execution time: mean = 36.818 us, total = 263.141 ms, Queueing time: mean = 264.727 us, max = 225.278 ms, min = 1.495 us, total = 1.892 s RayletLoadPulled - 4439 total (1 active), Execution time: mean = 246.810 us, total = 1.096 s, Queueing time: mean = 75.184 us, max = 1.351 ms, min = 10.621 us, total = 333.743 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4437 total (0 active), Execution time: mean = 76.038 us, total = 337.381 ms, Queueing time: mean = 95.629 us, max = 5.512 ms, min = 11.600 us, total = 424.304 ms NodeManagerService.grpc_client.GetResourceLoad - 4437 total (0 active), Execution time: mean = 1.388 ms, total = 6.160 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1787 total (0 active), Execution time: mean = 512.575 us, total = 915.972 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1787 total (0 active), Execution time: mean = 40.997 us, total = 73.261 ms, Queueing time: mean = 91.073 us, max = 601.063 us, min = 10.186 us, total = 162.747 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1775 total (0 active), Execution time: mean = 35.458 us, total = 62.937 ms, Queueing time: mean = 107.533 us, max = 1.496 ms, min = 14.438 us, total = 190.871 ms NodeInfoGcsService.grpc_server.CheckAlive - 1775 total (0 active), Execution time: mean = 531.375 us, total = 943.190 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1481 total (1 active), Execution time: mean = 10.426 us, total = 15.441 ms, Queueing time: mean = 73.372 us, max = 715.636 us, min = 7.259 us, total = 108.664 ms HealthCheck - 1478 total (0 active), Execution time: mean = 7.682 us, total = 11.353 ms, Queueing time: mean = 90.175 us, max = 288.247 us, min = 11.335 us, total = 133.278 ms GcsInMemoryStore.Get - 946 total (0 active), Execution time: mean = 21.509 us, total = 20.347 ms, Queueing time: mean = 12.954 us, max = 137.899 us, min = 2.227 us, total = 12.255 ms InternalKVGcsService.grpc_server.InternalKVGet - 945 total (0 active), Execution time: mean = 617.289 us, total = 583.338 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 945 total (0 active), Execution time: mean = 56.130 us, total = 53.043 ms, Queueing time: mean = 166.014 us, max = 22.982 ms, min = 9.184 us, total = 156.883 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 886 total (0 active), Execution time: mean = 622.269 us, total = 551.330 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 886 total (0 active), Execution time: mean = 91.372 us, total = 80.955 ms, Queueing time: mean = 106.178 us, max = 235.695 us, min = 15.511 us, total = 94.073 ms GCSServer.deadline_timer.debug_state_dump - 444 total (1 active), Execution time: mean = 1.385 ms, total = 614.854 ms, Queueing time: mean = 17.330 us, max = 166.240 us, min = -0.023 s, total = 7.694 ms GCSServer.deadline_timer.debug_state_event_stats_print - 74 total (1 active, 1 running), Execution time: mean = 1.556 ms, total = 115.180 ms, Queueing time: mean = 56.864 us, max = 137.733 us, min = 14.993 us, total = 4.208 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:26:15,308 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 204999 total (1 active) Queueing time: mean = 43.212 us, max = 4.613 ms, min = 1.891 us, total = 8.858 s Execution time: mean = 225.621 us, total = 46.252 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 93177 total (0 active), Execution time: mean = 467.811 us, total = 43.589 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 93177 total (0 active), Execution time: mean = 23.758 us, total = 2.214 s, Queueing time: mean = 92.662 us, max = 4.613 ms, min = 2.005 us, total = 8.634 s event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 24.799 us, total = 440.303 ms, Queueing time: mean = 8.752 us, max = 395.058 us, min = 1.891 us, total = 155.400 ms GcsTaskManager.GcJobSummary - 889 total (1 active), Execution time: mean = 9.877 us, total = 8.781 ms, Queueing time: mean = 77.634 us, max = 3.642 ms, min = 12.167 us, total = 69.017 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:26:15,309 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21891 total (25 active) Queueing time: mean = 16.420 us, max = 600.706 us, min = 2.413 us, total = 359.454 ms Execution time: mean = 4.600 s, total = 100688.534 s Event stats: event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 26.185 us, total = 464.921 ms, Queueing time: mean = 9.454 us, max = 600.706 us, min = 2.413 us, total = 167.849 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1110 total (0 active), Execution time: mean = 38.165 us, total = 42.363 ms, Queueing time: mean = 89.880 us, max = 408.208 us, min = 5.417 us, total = 99.767 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1110 total (24 active), Execution time: mean = 90.709 s, total = 100687.356 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 925 total (0 active), Execution time: mean = 104.670 us, total = 96.820 ms, Queueing time: mean = 95.938 us, max = 236.271 us, min = 11.318 us, total = 88.743 ms InternalPubSubGcsService.grpc_server.GcsPublish - 925 total (0 active), Execution time: mean = 573.317 us, total = 530.318 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 509.726 us, total = 7.646 ms, Queueing time: mean = 73.593 us, max = 114.456 us, min = 34.746 us, total = 1.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:26:15,309 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17801 total (0 active) Queueing time: mean = 10.392 us, max = 2.505 ms, min = 563.000 ns, total = 184.982 ms Execution time: mean = 27.175 us, total = 483.736 ms Event stats: event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 27.038 us, total = 480.051 ms, Queueing time: mean = 10.275 us, max = 2.505 ms, min = 1.873 us, total = 182.425 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:27:15,309 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 898 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 938 - cumulative published bytes: 746658 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:27:15,310 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 54294 total (4 active) Queueing time: mean = 78.181 us, max = 225.583 ms, min = -0.023 s, total = 4.245 s Execution time: mean = 233.429 us, total = 12.674 s Event stats: event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 25.891 us, total = 465.915 ms, Queueing time: mean = 10.099 us, max = 6.895 ms, min = 1.268 us, total = 181.732 ms GcsInMemoryStore.Put - 7243 total (0 active), Execution time: mean = 36.400 us, total = 263.644 ms, Queueing time: mean = 261.831 us, max = 225.278 ms, min = 1.495 us, total = 1.896 s RayletLoadPulled - 4499 total (1 active), Execution time: mean = 246.902 us, total = 1.111 s, Queueing time: mean = 75.312 us, max = 1.351 ms, min = 10.621 us, total = 338.827 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4497 total (0 active), Execution time: mean = 76.097 us, total = 342.209 ms, Queueing time: mean = 95.838 us, max = 5.512 ms, min = 11.600 us, total = 430.985 ms NodeManagerService.grpc_client.GetResourceLoad - 4497 total (0 active), Execution time: mean = 1.389 ms, total = 6.248 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1811 total (0 active), Execution time: mean = 513.811 us, total = 930.512 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1811 total (0 active), Execution time: mean = 40.980 us, total = 74.214 ms, Queueing time: mean = 91.390 us, max = 601.063 us, min = 10.186 us, total = 165.507 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1799 total (0 active), Execution time: mean = 35.455 us, total = 63.784 ms, Queueing time: mean = 107.835 us, max = 1.496 ms, min = 14.438 us, total = 193.995 ms NodeInfoGcsService.grpc_server.CheckAlive - 1799 total (0 active), Execution time: mean = 532.343 us, total = 957.685 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1501 total (1 active), Execution time: mean = 10.512 us, total = 15.778 ms, Queueing time: mean = 73.432 us, max = 715.636 us, min = 7.259 us, total = 110.221 ms HealthCheck - 1498 total (0 active), Execution time: mean = 7.688 us, total = 11.517 ms, Queueing time: mean = 90.377 us, max = 288.247 us, min = 11.335 us, total = 135.384 ms GcsInMemoryStore.Get - 958 total (0 active), Execution time: mean = 21.485 us, total = 20.583 ms, Queueing time: mean = 12.956 us, max = 137.899 us, min = 2.227 us, total = 12.412 ms InternalKVGcsService.grpc_server.InternalKVGet - 957 total (0 active), Execution time: mean = 617.802 us, total = 591.237 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 957 total (0 active), Execution time: mean = 56.114 us, total = 53.701 ms, Queueing time: mean = 165.433 us, max = 22.982 ms, min = 9.184 us, total = 158.319 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 898 total (0 active), Execution time: mean = 624.030 us, total = 560.379 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 898 total (0 active), Execution time: mean = 91.963 us, total = 82.583 ms, Queueing time: mean = 106.565 us, max = 235.695 us, min = 15.511 us, total = 95.695 ms GCSServer.deadline_timer.debug_state_dump - 450 total (1 active), Execution time: mean = 1.386 ms, total = 623.631 ms, Queueing time: mean = 18.165 us, max = 166.240 us, min = -0.023 s, total = 8.174 ms GCSServer.deadline_timer.debug_state_event_stats_print - 75 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 116.831 ms, Queueing time: mean = 57.005 us, max = 137.733 us, min = 14.993 us, total = 4.275 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:27:15,310 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 207771 total (1 active) Queueing time: mean = 43.569 us, max = 58.960 ms, min = 1.891 us, total = 9.052 s Execution time: mean = 226.331 us, total = 47.025 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 94437 total (0 active), Execution time: mean = 469.253 us, total = 44.315 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 94437 total (0 active), Execution time: mean = 23.792 us, total = 2.247 s, Queueing time: mean = 93.444 us, max = 58.960 ms, min = 2.005 us, total = 8.825 s event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 25.254 us, total = 454.438 ms, Queueing time: mean = 8.767 us, max = 395.058 us, min = 1.891 us, total = 157.766 ms GcsTaskManager.GcJobSummary - 901 total (1 active), Execution time: mean = 9.880 us, total = 8.902 ms, Queueing time: mean = 77.606 us, max = 3.642 ms, min = 12.167 us, total = 69.923 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:27:15,310 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22194 total (25 active) Queueing time: mean = 16.456 us, max = 600.706 us, min = 2.413 us, total = 365.216 ms Execution time: mean = 4.621 s, total = 102548.735 s Event stats: event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 26.184 us, total = 471.178 ms, Queueing time: mean = 9.452 us, max = 600.706 us, min = 2.413 us, total = 170.097 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1128 total (0 active), Execution time: mean = 38.267 us, total = 43.165 ms, Queueing time: mean = 90.238 us, max = 408.208 us, min = 5.417 us, total = 101.789 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1128 total (24 active), Execution time: mean = 90.911 s, total = 102547.540 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 938 total (0 active), Execution time: mean = 104.912 us, total = 98.408 ms, Queueing time: mean = 96.091 us, max = 236.271 us, min = 11.318 us, total = 90.134 ms InternalPubSubGcsService.grpc_server.GcsPublish - 938 total (0 active), Execution time: mean = 573.934 us, total = 538.350 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 496.077 us, total = 7.937 ms, Queueing time: mean = 75.375 us, max = 114.456 us, min = 34.746 us, total = 1.206 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:27:15,310 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18041 total (0 active) Queueing time: mean = 10.386 us, max = 2.505 ms, min = 563.000 ns, total = 187.370 ms Execution time: mean = 27.172 us, total = 490.211 ms Event stats: event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 27.037 us, total = 486.527 ms, Queueing time: mean = 10.270 us, max = 2.505 ms, min = 1.873 us, total = 184.814 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:28:15,310 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 910 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 950 - cumulative published bytes: 756486 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:28:15,311 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 55013 total (4 active) Queueing time: mean = 77.695 us, max = 225.583 ms, min = -0.023 s, total = 4.274 s Execution time: mean = 233.308 us, total = 12.835 s Event stats: event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 25.906 us, total = 472.402 ms, Queueing time: mean = 10.097 us, max = 6.895 ms, min = 1.268 us, total = 184.119 ms GcsInMemoryStore.Put - 7339 total (0 active), Execution time: mean = 35.993 us, total = 264.155 ms, Queueing time: mean = 258.946 us, max = 225.278 ms, min = 1.495 us, total = 1.900 s RayletLoadPulled - 4559 total (1 active), Execution time: mean = 246.933 us, total = 1.126 s, Queueing time: mean = 75.400 us, max = 1.351 ms, min = 10.621 us, total = 343.747 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4557 total (0 active), Execution time: mean = 76.131 us, total = 346.929 ms, Queueing time: mean = 95.993 us, max = 5.512 ms, min = 11.600 us, total = 437.441 ms NodeManagerService.grpc_client.GetResourceLoad - 4557 total (0 active), Execution time: mean = 1.389 ms, total = 6.329 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1835 total (0 active), Execution time: mean = 513.484 us, total = 942.242 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1835 total (0 active), Execution time: mean = 41.019 us, total = 75.271 ms, Queueing time: mean = 91.433 us, max = 601.063 us, min = 10.186 us, total = 167.779 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1823 total (0 active), Execution time: mean = 35.606 us, total = 64.910 ms, Queueing time: mean = 107.909 us, max = 1.496 ms, min = 14.438 us, total = 196.717 ms NodeInfoGcsService.grpc_server.CheckAlive - 1823 total (0 active), Execution time: mean = 532.817 us, total = 971.326 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1521 total (1 active), Execution time: mean = 10.513 us, total = 15.990 ms, Queueing time: mean = 73.277 us, max = 715.636 us, min = 7.259 us, total = 111.455 ms HealthCheck - 1518 total (0 active), Execution time: mean = 7.686 us, total = 11.667 ms, Queueing time: mean = 90.685 us, max = 288.247 us, min = 11.335 us, total = 137.660 ms GcsInMemoryStore.Get - 970 total (0 active), Execution time: mean = 21.483 us, total = 20.838 ms, Queueing time: mean = 12.945 us, max = 137.899 us, min = 2.227 us, total = 12.556 ms InternalKVGcsService.grpc_server.InternalKVGet - 969 total (0 active), Execution time: mean = 616.641 us, total = 597.525 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 969 total (0 active), Execution time: mean = 56.225 us, total = 54.482 ms, Queueing time: mean = 164.654 us, max = 22.982 ms, min = 9.184 us, total = 159.549 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 910 total (0 active), Execution time: mean = 623.739 us, total = 567.602 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 910 total (0 active), Execution time: mean = 92.036 us, total = 83.753 ms, Queueing time: mean = 106.743 us, max = 235.695 us, min = 15.511 us, total = 97.136 ms GCSServer.deadline_timer.debug_state_dump - 456 total (1 active), Execution time: mean = 1.387 ms, total = 632.348 ms, Queueing time: mean = 18.791 us, max = 166.240 us, min = -0.023 s, total = 8.569 ms GCSServer.deadline_timer.debug_state_event_stats_print - 76 total (1 active, 1 running), Execution time: mean = 1.555 ms, total = 118.171 ms, Queueing time: mean = 57.360 us, max = 137.733 us, min = 14.993 us, total = 4.359 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:28:15,312 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 210543 total (1 active) Queueing time: mean = 43.529 us, max = 58.960 ms, min = 1.891 us, total = 9.165 s Execution time: mean = 226.171 us, total = 47.619 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 95697 total (0 active), Execution time: mean = 468.897 us, total = 44.872 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 95697 total (0 active), Execution time: mean = 23.790 us, total = 2.277 s, Queueing time: mean = 93.356 us, max = 58.960 ms, min = 2.005 us, total = 8.934 s event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 25.288 us, total = 461.126 ms, Queueing time: mean = 8.779 us, max = 395.058 us, min = 1.891 us, total = 160.088 ms GcsTaskManager.GcJobSummary - 913 total (1 active), Execution time: mean = 9.894 us, total = 9.034 ms, Queueing time: mean = 77.586 us, max = 3.642 ms, min = 12.167 us, total = 70.836 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:28:15,312 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22482 total (25 active) Queueing time: mean = 16.428 us, max = 600.706 us, min = 2.413 us, total = 369.331 ms Execution time: mean = 4.564 s, total = 102608.885 s Event stats: event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 26.147 us, total = 476.800 ms, Queueing time: mean = 9.439 us, max = 600.706 us, min = 2.413 us, total = 172.127 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1140 total (0 active), Execution time: mean = 38.363 us, total = 43.734 ms, Queueing time: mean = 90.081 us, max = 408.208 us, min = 5.417 us, total = 102.693 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1140 total (24 active), Execution time: mean = 90.007 s, total = 102607.675 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 950 total (0 active), Execution time: mean = 104.772 us, total = 99.533 ms, Queueing time: mean = 96.119 us, max = 236.271 us, min = 11.318 us, total = 91.313 ms InternalPubSubGcsService.grpc_server.GcsPublish - 950 total (0 active), Execution time: mean = 573.704 us, total = 545.019 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 496.077 us, total = 7.937 ms, Queueing time: mean = 75.375 us, max = 114.456 us, min = 34.746 us, total = 1.206 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:28:15,312 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18281 total (0 active) Queueing time: mean = 10.383 us, max = 2.505 ms, min = 563.000 ns, total = 189.805 ms Execution time: mean = 27.176 us, total = 496.808 ms Event stats: event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 27.043 us, total = 493.124 ms, Queueing time: mean = 10.269 us, max = 2.505 ms, min = 1.873 us, total = 187.249 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:29:15,312 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 922 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 963 - cumulative published bytes: 766348 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:29:15,313 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 55731 total (4 active) Queueing time: mean = 77.213 us, max = 225.583 ms, min = -0.023 s, total = 4.303 s Execution time: mean = 233.108 us, total = 12.991 s Event stats: event_loop_lag_probe - 18474 total (0 active), Execution time: mean = 25.911 us, total = 478.686 ms, Queueing time: mean = 10.097 us, max = 6.895 ms, min = 1.268 us, total = 186.539 ms GcsInMemoryStore.Put - 7435 total (0 active), Execution time: mean = 35.594 us, total = 264.645 ms, Queueing time: mean = 256.167 us, max = 225.278 ms, min = 1.495 us, total = 1.905 s RayletLoadPulled - 4619 total (1 active), Execution time: mean = 246.896 us, total = 1.140 s, Queueing time: mean = 75.456 us, max = 1.351 ms, min = 10.621 us, total = 348.531 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4617 total (0 active), Execution time: mean = 76.137 us, total = 351.526 ms, Queueing time: mean = 96.050 us, max = 5.512 ms, min = 11.600 us, total = 443.465 ms NodeManagerService.grpc_client.GetResourceLoad - 4617 total (0 active), Execution time: mean = 1.389 ms, total = 6.412 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1859 total (0 active), Execution time: mean = 512.430 us, total = 952.606 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1859 total (0 active), Execution time: mean = 40.976 us, total = 76.175 ms, Queueing time: mean = 91.531 us, max = 601.063 us, min = 10.186 us, total = 170.156 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1847 total (0 active), Execution time: mean = 35.561 us, total = 65.681 ms, Queueing time: mean = 107.788 us, max = 1.496 ms, min = 14.438 us, total = 199.085 ms NodeInfoGcsService.grpc_server.CheckAlive - 1847 total (0 active), Execution time: mean = 531.645 us, total = 981.948 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1541 total (1 active), Execution time: mean = 10.495 us, total = 16.172 ms, Queueing time: mean = 73.204 us, max = 715.636 us, min = 7.259 us, total = 112.808 ms HealthCheck - 1538 total (0 active), Execution time: mean = 7.675 us, total = 11.804 ms, Queueing time: mean = 90.813 us, max = 288.247 us, min = 11.335 us, total = 139.670 ms GcsInMemoryStore.Get - 982 total (0 active), Execution time: mean = 21.484 us, total = 21.097 ms, Queueing time: mean = 12.944 us, max = 137.899 us, min = 2.227 us, total = 12.711 ms InternalKVGcsService.grpc_server.InternalKVGet - 981 total (0 active), Execution time: mean = 615.393 us, total = 603.701 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 981 total (0 active), Execution time: mean = 56.427 us, total = 55.354 ms, Queueing time: mean = 163.872 us, max = 22.982 ms, min = 9.184 us, total = 160.758 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 922 total (0 active), Execution time: mean = 622.951 us, total = 574.360 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 922 total (0 active), Execution time: mean = 91.941 us, total = 84.769 ms, Queueing time: mean = 106.940 us, max = 235.695 us, min = 15.511 us, total = 98.598 ms GCSServer.deadline_timer.debug_state_dump - 462 total (1 active), Execution time: mean = 1.386 ms, total = 640.407 ms, Queueing time: mean = 19.520 us, max = 166.240 us, min = -0.023 s, total = 9.018 ms GCSServer.deadline_timer.debug_state_event_stats_print - 77 total (1 active, 1 running), Execution time: mean = 1.556 ms, total = 119.824 ms, Queueing time: mean = 57.492 us, max = 137.733 us, min = 14.993 us, total = 4.427 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:29:15,313 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 213315 total (1 active) Queueing time: mean = 43.321 us, max = 58.960 ms, min = 1.891 us, total = 9.241 s Execution time: mean = 225.551 us, total = 48.113 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 96957 total (0 active), Execution time: mean = 467.594 us, total = 45.336 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 96957 total (0 active), Execution time: mean = 23.724 us, total = 2.300 s, Queueing time: mean = 92.899 us, max = 58.960 ms, min = 2.005 us, total = 9.007 s event_loop_lag_probe - 18475 total (0 active), Execution time: mean = 25.304 us, total = 467.499 ms, Queueing time: mean = 8.781 us, max = 395.058 us, min = 1.891 us, total = 162.221 ms GcsTaskManager.GcJobSummary - 925 total (1 active), Execution time: mean = 9.874 us, total = 9.133 ms, Queueing time: mean = 77.390 us, max = 3.642 ms, min = 12.167 us, total = 71.586 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:29:15,314 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22774 total (25 active) Queueing time: mean = 16.380 us, max = 600.706 us, min = 2.413 us, total = 373.039 ms Execution time: mean = 4.508 s, total = 102669.056 s Event stats: event_loop_lag_probe - 18475 total (0 active), Execution time: mean = 26.126 us, total = 482.669 ms, Queueing time: mean = 9.430 us, max = 600.706 us, min = 2.413 us, total = 174.222 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1153 total (0 active), Execution time: mean = 38.408 us, total = 44.285 ms, Queueing time: mean = 89.669 us, max = 408.208 us, min = 5.417 us, total = 103.389 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1153 total (24 active), Execution time: mean = 89.044 s, total = 102667.833 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 963 total (0 active), Execution time: mean = 104.898 us, total = 101.017 ms, Queueing time: mean = 95.774 us, max = 236.271 us, min = 11.318 us, total = 92.230 ms InternalPubSubGcsService.grpc_server.GcsPublish - 963 total (0 active), Execution time: mean = 572.471 us, total = 551.289 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 496.077 us, total = 7.937 ms, Queueing time: mean = 75.375 us, max = 114.456 us, min = 34.746 us, total = 1.206 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:29:15,314 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18521 total (0 active) Queueing time: mean = 10.369 us, max = 2.505 ms, min = 563.000 ns, total = 192.043 ms Execution time: mean = 27.149 us, total = 502.831 ms Event stats: event_loop_lag_probe - 18475 total (0 active), Execution time: mean = 27.017 us, total = 499.147 ms, Queueing time: mean = 10.256 us, max = 2.505 ms, min = 1.873 us, total = 189.487 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:30:15,314 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 934 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 975 - cumulative published bytes: 776176 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:30:15,315 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 56450 total (4 active) Queueing time: mean = 76.752 us, max = 225.583 ms, min = -0.023 s, total = 4.333 s Execution time: mean = 232.819 us, total = 13.143 s Event stats: event_loop_lag_probe - 18714 total (0 active), Execution time: mean = 25.925 us, total = 485.155 ms, Queueing time: mean = 10.096 us, max = 6.895 ms, min = 1.268 us, total = 188.933 ms GcsInMemoryStore.Put - 7531 total (0 active), Execution time: mean = 35.209 us, total = 265.157 ms, Queueing time: mean = 253.482 us, max = 225.278 ms, min = 1.495 us, total = 1.909 s RayletLoadPulled - 4679 total (1 active), Execution time: mean = 246.908 us, total = 1.155 s, Queueing time: mean = 75.390 us, max = 1.351 ms, min = 10.621 us, total = 352.750 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4677 total (0 active), Execution time: mean = 76.099 us, total = 355.914 ms, Queueing time: mean = 96.111 us, max = 5.512 ms, min = 11.600 us, total = 449.510 ms NodeManagerService.grpc_client.GetResourceLoad - 4677 total (0 active), Execution time: mean = 1.386 ms, total = 6.484 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1883 total (0 active), Execution time: mean = 512.231 us, total = 964.532 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1883 total (0 active), Execution time: mean = 41.034 us, total = 77.267 ms, Queueing time: mean = 91.790 us, max = 601.063 us, min = 10.186 us, total = 172.841 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1871 total (0 active), Execution time: mean = 35.519 us, total = 66.456 ms, Queueing time: mean = 107.852 us, max = 1.496 ms, min = 14.438 us, total = 201.790 ms NodeInfoGcsService.grpc_server.CheckAlive - 1871 total (0 active), Execution time: mean = 531.153 us, total = 993.787 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1561 total (1 active), Execution time: mean = 10.501 us, total = 16.393 ms, Queueing time: mean = 73.305 us, max = 715.636 us, min = 7.259 us, total = 114.429 ms HealthCheck - 1558 total (0 active), Execution time: mean = 7.664 us, total = 11.941 ms, Queueing time: mean = 90.985 us, max = 288.247 us, min = 11.335 us, total = 141.755 ms GcsInMemoryStore.Get - 994 total (0 active), Execution time: mean = 21.470 us, total = 21.341 ms, Queueing time: mean = 12.940 us, max = 137.899 us, min = 2.227 us, total = 12.862 ms InternalKVGcsService.grpc_server.InternalKVGet - 993 total (0 active), Execution time: mean = 614.259 us, total = 609.959 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 993 total (0 active), Execution time: mean = 56.519 us, total = 56.124 ms, Queueing time: mean = 163.074 us, max = 22.982 ms, min = 9.184 us, total = 161.933 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 934 total (0 active), Execution time: mean = 622.272 us, total = 581.202 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 934 total (0 active), Execution time: mean = 91.837 us, total = 85.776 ms, Queueing time: mean = 107.244 us, max = 235.695 us, min = 15.511 us, total = 100.166 ms GCSServer.deadline_timer.debug_state_dump - 468 total (1 active), Execution time: mean = 1.389 ms, total = 649.819 ms, Queueing time: mean = 20.167 us, max = 166.240 us, min = -0.023 s, total = 9.438 ms GCSServer.deadline_timer.debug_state_event_stats_print - 78 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 121.532 ms, Queueing time: mean = 57.556 us, max = 137.733 us, min = 14.993 us, total = 4.489 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:30:15,315 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 216087 total (1 active) Queueing time: mean = 43.162 us, max = 58.960 ms, min = 1.891 us, total = 9.327 s Execution time: mean = 224.971 us, total = 48.613 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 98217 total (0 active), Execution time: mean = 466.385 us, total = 45.807 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 98217 total (0 active), Execution time: mean = 23.661 us, total = 2.324 s, Queueing time: mean = 92.555 us, max = 58.960 ms, min = 2.005 us, total = 9.090 s event_loop_lag_probe - 18715 total (0 active), Execution time: mean = 25.284 us, total = 473.186 ms, Queueing time: mean = 8.772 us, max = 395.058 us, min = 1.891 us, total = 164.165 ms GcsTaskManager.GcJobSummary - 937 total (1 active), Execution time: mean = 9.838 us, total = 9.219 ms, Queueing time: mean = 76.985 us, max = 3.642 ms, min = 12.167 us, total = 72.135 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:30:15,315 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23062 total (25 active) Queueing time: mean = 16.335 us, max = 600.706 us, min = 1.622 us, total = 376.710 ms Execution time: mean = 4.454 s, total = 102729.185 s Event stats: event_loop_lag_probe - 18715 total (0 active), Execution time: mean = 26.066 us, total = 487.828 ms, Queueing time: mean = 9.409 us, max = 600.706 us, min = 1.622 us, total = 176.086 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1165 total (0 active), Execution time: mean = 38.317 us, total = 44.640 ms, Queueing time: mean = 89.499 us, max = 408.208 us, min = 5.417 us, total = 104.267 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1165 total (24 active), Execution time: mean = 88.178 s, total = 102727.950 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 975 total (0 active), Execution time: mean = 104.525 us, total = 101.912 ms, Queueing time: mean = 95.549 us, max = 236.271 us, min = 11.318 us, total = 93.160 ms InternalPubSubGcsService.grpc_server.GcsPublish - 975 total (0 active), Execution time: mean = 570.903 us, total = 556.631 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 496.077 us, total = 7.937 ms, Queueing time: mean = 75.375 us, max = 114.456 us, min = 34.746 us, total = 1.206 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:30:15,315 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18761 total (0 active) Queueing time: mean = 10.342 us, max = 2.505 ms, min = 563.000 ns, total = 194.029 ms Execution time: mean = 27.085 us, total = 508.149 ms Event stats: event_loop_lag_probe - 18715 total (0 active), Execution time: mean = 26.955 us, total = 504.465 ms, Queueing time: mean = 10.231 us, max = 2.505 ms, min = 1.873 us, total = 191.473 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:31:15,316 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 946 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 987 - cumulative published bytes: 786004 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:31:15,316 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 57169 total (4 active) Queueing time: mean = 76.335 us, max = 225.583 ms, min = -0.023 s, total = 4.364 s Execution time: mean = 232.664 us, total = 13.301 s Event stats: event_loop_lag_probe - 18954 total (0 active), Execution time: mean = 25.945 us, total = 491.759 ms, Queueing time: mean = 10.096 us, max = 6.895 ms, min = 1.268 us, total = 191.369 ms GcsInMemoryStore.Put - 7627 total (0 active), Execution time: mean = 34.839 us, total = 265.715 ms, Queueing time: mean = 250.901 us, max = 225.278 ms, min = 1.495 us, total = 1.914 s RayletLoadPulled - 4739 total (1 active), Execution time: mean = 246.956 us, total = 1.170 s, Queueing time: mean = 75.501 us, max = 1.351 ms, min = 10.621 us, total = 357.800 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4737 total (0 active), Execution time: mean = 76.124 us, total = 360.598 ms, Queueing time: mean = 96.329 us, max = 5.512 ms, min = 11.600 us, total = 456.309 ms NodeManagerService.grpc_client.GetResourceLoad - 4737 total (0 active), Execution time: mean = 1.386 ms, total = 6.565 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1907 total (0 active), Execution time: mean = 511.762 us, total = 975.930 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1907 total (0 active), Execution time: mean = 41.064 us, total = 78.309 ms, Queueing time: mean = 91.956 us, max = 601.063 us, min = 10.186 us, total = 175.360 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1895 total (0 active), Execution time: mean = 35.572 us, total = 67.410 ms, Queueing time: mean = 108.066 us, max = 1.496 ms, min = 14.438 us, total = 204.784 ms NodeInfoGcsService.grpc_server.CheckAlive - 1895 total (0 active), Execution time: mean = 530.759 us, total = 1.006 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1581 total (1 active), Execution time: mean = 10.501 us, total = 16.602 ms, Queueing time: mean = 73.367 us, max = 715.636 us, min = 7.259 us, total = 115.993 ms HealthCheck - 1578 total (0 active), Execution time: mean = 7.673 us, total = 12.109 ms, Queueing time: mean = 91.072 us, max = 288.247 us, min = 11.335 us, total = 143.711 ms GcsInMemoryStore.Get - 1006 total (0 active), Execution time: mean = 21.464 us, total = 21.593 ms, Queueing time: mean = 12.936 us, max = 137.899 us, min = 2.227 us, total = 13.014 ms InternalKVGcsService.grpc_server.InternalKVGet - 1005 total (0 active), Execution time: mean = 613.022 us, total = 616.087 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1005 total (0 active), Execution time: mean = 56.469 us, total = 56.751 ms, Queueing time: mean = 162.498 us, max = 22.982 ms, min = 9.184 us, total = 163.311 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 946 total (0 active), Execution time: mean = 622.031 us, total = 588.441 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 946 total (0 active), Execution time: mean = 91.726 us, total = 86.773 ms, Queueing time: mean = 107.418 us, max = 235.695 us, min = 15.511 us, total = 101.618 ms GCSServer.deadline_timer.debug_state_dump - 474 total (1 active), Execution time: mean = 1.389 ms, total = 658.539 ms, Queueing time: mean = 20.633 us, max = 166.240 us, min = -0.023 s, total = 9.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 79 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 123.133 ms, Queueing time: mean = 57.447 us, max = 137.733 us, min = 14.993 us, total = 4.538 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:31:15,317 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 218858 total (1 active) Queueing time: mean = 43.111 us, max = 58.960 ms, min = 1.891 us, total = 9.435 s Execution time: mean = 224.790 us, total = 49.197 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 99477 total (0 active), Execution time: mean = 465.994 us, total = 46.356 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 99477 total (0 active), Execution time: mean = 23.648 us, total = 2.352 s, Queueing time: mean = 92.441 us, max = 58.960 ms, min = 2.005 us, total = 9.196 s event_loop_lag_probe - 18954 total (0 active), Execution time: mean = 25.300 us, total = 479.535 ms, Queueing time: mean = 8.775 us, max = 395.058 us, min = 1.891 us, total = 166.325 ms GcsTaskManager.GcJobSummary - 949 total (1 active), Execution time: mean = 9.852 us, total = 9.350 ms, Queueing time: mean = 76.957 us, max = 3.642 ms, min = 12.167 us, total = 73.032 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:31:15,317 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23350 total (25 active) Queueing time: mean = 16.301 us, max = 600.706 us, min = 1.622 us, total = 380.631 ms Execution time: mean = 4.402 s, total = 102789.387 s Event stats: event_loop_lag_probe - 18955 total (0 active), Execution time: mean = 26.032 us, total = 493.439 ms, Queueing time: mean = 9.397 us, max = 600.706 us, min = 1.622 us, total = 178.125 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1177 total (0 active), Execution time: mean = 38.275 us, total = 45.049 ms, Queueing time: mean = 89.287 us, max = 408.208 us, min = 5.417 us, total = 105.090 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1177 total (24 active), Execution time: mean = 87.331 s, total = 102788.138 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 987 total (0 active), Execution time: mean = 104.430 us, total = 103.073 ms, Queueing time: mean = 95.460 us, max = 236.271 us, min = 11.318 us, total = 94.219 ms InternalPubSubGcsService.grpc_server.GcsPublish - 987 total (0 active), Execution time: mean = 570.376 us, total = 562.961 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 496.077 us, total = 7.937 ms, Queueing time: mean = 75.375 us, max = 114.456 us, min = 34.746 us, total = 1.206 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:31:15,317 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19001 total (0 active) Queueing time: mean = 10.330 us, max = 2.505 ms, min = 563.000 ns, total = 196.277 ms Execution time: mean = 27.066 us, total = 514.287 ms Event stats: event_loop_lag_probe - 18955 total (0 active), Execution time: mean = 26.938 us, total = 510.603 ms, Queueing time: mean = 10.220 us, max = 2.505 ms, min = 1.873 us, total = 193.721 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:32:15,317 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 958 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 999 - cumulative published bytes: 795832 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:32:15,318 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 57888 total (4 active) Queueing time: mean = 75.856 us, max = 225.583 ms, min = -0.023 s, total = 4.391 s Execution time: mean = 232.362 us, total = 13.451 s Event stats: event_loop_lag_probe - 19194 total (0 active), Execution time: mean = 25.936 us, total = 497.814 ms, Queueing time: mean = 10.099 us, max = 6.895 ms, min = 1.268 us, total = 193.839 ms GcsInMemoryStore.Put - 7723 total (0 active), Execution time: mean = 34.471 us, total = 266.217 ms, Queueing time: mean = 248.341 us, max = 225.278 ms, min = 1.495 us, total = 1.918 s RayletLoadPulled - 4799 total (1 active), Execution time: mean = 247.086 us, total = 1.186 s, Queueing time: mean = 75.429 us, max = 1.351 ms, min = 10.621 us, total = 361.982 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4797 total (0 active), Execution time: mean = 76.197 us, total = 365.519 ms, Queueing time: mean = 96.321 us, max = 5.512 ms, min = 11.600 us, total = 462.053 ms NodeManagerService.grpc_client.GetResourceLoad - 4797 total (0 active), Execution time: mean = 1.384 ms, total = 6.640 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1931 total (0 active), Execution time: mean = 510.805 us, total = 986.365 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1931 total (0 active), Execution time: mean = 41.072 us, total = 79.310 ms, Queueing time: mean = 91.892 us, max = 601.063 us, min = 10.186 us, total = 177.444 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1919 total (0 active), Execution time: mean = 35.570 us, total = 68.258 ms, Queueing time: mean = 107.864 us, max = 1.496 ms, min = 14.438 us, total = 206.991 ms NodeInfoGcsService.grpc_server.CheckAlive - 1919 total (0 active), Execution time: mean = 529.695 us, total = 1.016 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1601 total (1 active), Execution time: mean = 10.593 us, total = 16.960 ms, Queueing time: mean = 73.324 us, max = 715.636 us, min = 7.259 us, total = 117.392 ms HealthCheck - 1598 total (0 active), Execution time: mean = 7.684 us, total = 12.278 ms, Queueing time: mean = 91.003 us, max = 288.247 us, min = 11.335 us, total = 145.422 ms GcsInMemoryStore.Get - 1018 total (0 active), Execution time: mean = 21.458 us, total = 21.844 ms, Queueing time: mean = 12.927 us, max = 137.899 us, min = 2.227 us, total = 13.160 ms InternalKVGcsService.grpc_server.InternalKVGet - 1017 total (0 active), Execution time: mean = 611.776 us, total = 622.176 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1017 total (0 active), Execution time: mean = 56.444 us, total = 57.404 ms, Queueing time: mean = 161.741 us, max = 22.982 ms, min = 9.184 us, total = 164.490 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 958 total (0 active), Execution time: mean = 621.506 us, total = 595.403 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 958 total (0 active), Execution time: mean = 91.801 us, total = 87.945 ms, Queueing time: mean = 107.493 us, max = 235.695 us, min = 15.511 us, total = 102.978 ms GCSServer.deadline_timer.debug_state_dump - 480 total (1 active), Execution time: mean = 1.387 ms, total = 665.947 ms, Queueing time: mean = 21.115 us, max = 166.240 us, min = -0.023 s, total = 10.135 ms GCSServer.deadline_timer.debug_state_event_stats_print - 80 total (1 active, 1 running), Execution time: mean = 1.560 ms, total = 124.774 ms, Queueing time: mean = 57.153 us, max = 137.733 us, min = 14.993 us, total = 4.572 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:32:15,318 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 221630 total (1 active) Queueing time: mean = 42.980 us, max = 58.960 ms, min = 1.891 us, total = 9.526 s Execution time: mean = 224.293 us, total = 49.710 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 100737 total (0 active), Execution time: mean = 464.936 us, total = 46.836 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 100737 total (0 active), Execution time: mean = 23.608 us, total = 2.378 s, Queueing time: mean = 92.152 us, max = 58.960 ms, min = 2.005 us, total = 9.283 s event_loop_lag_probe - 19194 total (0 active), Execution time: mean = 25.321 us, total = 486.013 ms, Queueing time: mean = 8.777 us, max = 395.058 us, min = 1.891 us, total = 168.472 ms GcsTaskManager.GcJobSummary - 961 total (1 active), Execution time: mean = 9.862 us, total = 9.477 ms, Queueing time: mean = 76.900 us, max = 3.642 ms, min = 12.167 us, total = 73.900 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:32:15,319 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23675 total (25 active) Queueing time: mean = 16.273 us, max = 600.706 us, min = 1.622 us, total = 385.265 ms Execution time: mean = 4.800 s, total = 113649.519 s Event stats: event_loop_lag_probe - 19195 total (0 active), Execution time: mean = 26.003 us, total = 499.118 ms, Queueing time: mean = 9.383 us, max = 600.706 us, min = 1.622 us, total = 180.109 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1207 total (0 active), Execution time: mean = 37.882 us, total = 45.724 ms, Queueing time: mean = 88.331 us, max = 408.208 us, min = 5.417 us, total = 106.616 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1207 total (24 active), Execution time: mean = 94.158 s, total = 113648.257 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 999 total (0 active), Execution time: mean = 104.332 us, total = 104.228 ms, Queueing time: mean = 95.404 us, max = 236.271 us, min = 11.318 us, total = 95.308 ms InternalPubSubGcsService.grpc_server.GcsPublish - 999 total (0 active), Execution time: mean = 569.802 us, total = 569.232 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 490.117 us, total = 8.332 ms, Queueing time: mean = 72.999 us, max = 114.456 us, min = 34.746 us, total = 1.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:32:15,319 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19241 total (0 active) Queueing time: mean = 10.310 us, max = 2.505 ms, min = 563.000 ns, total = 198.365 ms Execution time: mean = 27.020 us, total = 519.888 ms Event stats: event_loop_lag_probe - 19195 total (0 active), Execution time: mean = 26.893 us, total = 516.204 ms, Queueing time: mean = 10.201 us, max = 2.505 ms, min = 1.873 us, total = 195.809 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:33:15,319 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 970 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1012 - cumulative published bytes: 805694 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:33:15,319 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 58607 total (4 active) Queueing time: mean = 75.390 us, max = 225.583 ms, min = -0.023 s, total = 4.418 s Execution time: mean = 231.631 us, total = 13.575 s Event stats: event_loop_lag_probe - 19434 total (0 active), Execution time: mean = 25.906 us, total = 503.466 ms, Queueing time: mean = 10.097 us, max = 6.895 ms, min = 1.268 us, total = 196.224 ms GcsInMemoryStore.Put - 7819 total (0 active), Execution time: mean = 34.108 us, total = 266.691 ms, Queueing time: mean = 245.854 us, max = 225.278 ms, min = 1.495 us, total = 1.922 s RayletLoadPulled - 4859 total (1 active), Execution time: mean = 246.869 us, total = 1.200 s, Queueing time: mean = 75.344 us, max = 1.351 ms, min = 10.621 us, total = 366.096 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4857 total (0 active), Execution time: mean = 76.086 us, total = 369.548 ms, Queueing time: mean = 96.336 us, max = 5.512 ms, min = 11.600 us, total = 467.902 ms NodeManagerService.grpc_client.GetResourceLoad - 4857 total (0 active), Execution time: mean = 1.379 ms, total = 6.700 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1955 total (0 active), Execution time: mean = 509.219 us, total = 995.523 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1955 total (0 active), Execution time: mean = 40.987 us, total = 80.129 ms, Queueing time: mean = 91.894 us, max = 601.063 us, min = 10.186 us, total = 179.652 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1943 total (0 active), Execution time: mean = 35.470 us, total = 68.918 ms, Queueing time: mean = 107.688 us, max = 1.496 ms, min = 14.438 us, total = 209.239 ms NodeInfoGcsService.grpc_server.CheckAlive - 1943 total (0 active), Execution time: mean = 527.604 us, total = 1.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1621 total (1 active), Execution time: mean = 10.564 us, total = 17.125 ms, Queueing time: mean = 73.165 us, max = 715.636 us, min = 7.259 us, total = 118.600 ms HealthCheck - 1618 total (0 active), Execution time: mean = 7.678 us, total = 12.423 ms, Queueing time: mean = 91.029 us, max = 288.247 us, min = 11.335 us, total = 147.286 ms GcsInMemoryStore.Get - 1030 total (0 active), Execution time: mean = 21.411 us, total = 22.054 ms, Queueing time: mean = 12.913 us, max = 137.899 us, min = 2.227 us, total = 13.301 ms InternalKVGcsService.grpc_server.InternalKVGet - 1029 total (0 active), Execution time: mean = 609.573 us, total = 627.251 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1029 total (0 active), Execution time: mean = 56.320 us, total = 57.954 ms, Queueing time: mean = 160.911 us, max = 22.982 ms, min = 9.184 us, total = 165.577 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 970 total (0 active), Execution time: mean = 620.241 us, total = 601.633 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 970 total (0 active), Execution time: mean = 91.551 us, total = 88.805 ms, Queueing time: mean = 107.625 us, max = 235.695 us, min = 15.511 us, total = 104.396 ms GCSServer.deadline_timer.debug_state_dump - 486 total (1 active), Execution time: mean = 1.383 ms, total = 672.321 ms, Queueing time: mean = 21.345 us, max = 166.240 us, min = -0.023 s, total = 10.374 ms GCSServer.deadline_timer.debug_state_event_stats_print - 81 total (1 active, 1 running), Execution time: mean = 1.561 ms, total = 126.475 ms, Queueing time: mean = 57.112 us, max = 137.733 us, min = 14.993 us, total = 4.626 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:33:15,319 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 224402 total (1 active) Queueing time: mean = 42.634 us, max = 58.960 ms, min = 1.891 us, total = 9.567 s Execution time: mean = 223.014 us, total = 50.045 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 101997 total (0 active), Execution time: mean = 462.239 us, total = 47.147 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 101997 total (0 active), Execution time: mean = 23.492 us, total = 2.396 s, Queueing time: mean = 91.394 us, max = 58.960 ms, min = 2.005 us, total = 9.322 s event_loop_lag_probe - 19434 total (0 active), Execution time: mean = 25.327 us, total = 492.203 ms, Queueing time: mean = 8.781 us, max = 395.058 us, min = 1.891 us, total = 170.655 ms GcsTaskManager.GcJobSummary - 973 total (1 active), Execution time: mean = 9.849 us, total = 9.583 ms, Queueing time: mean = 76.604 us, max = 3.642 ms, min = 12.167 us, total = 74.536 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:33:15,319 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23967 total (25 active) Queueing time: mean = 16.201 us, max = 600.706 us, min = 1.622 us, total = 388.301 ms Execution time: mean = 4.744 s, total = 113709.732 s Event stats: event_loop_lag_probe - 19435 total (0 active), Execution time: mean = 25.980 us, total = 504.917 ms, Queueing time: mean = 9.371 us, max = 600.706 us, min = 1.622 us, total = 182.132 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1220 total (0 active), Execution time: mean = 37.810 us, total = 46.129 ms, Queueing time: mean = 87.753 us, max = 408.208 us, min = 5.417 us, total = 107.059 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1220 total (24 active), Execution time: mean = 93.204 s, total = 113708.456 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1012 total (0 active), Execution time: mean = 104.218 us, total = 105.469 ms, Queueing time: mean = 94.741 us, max = 236.271 us, min = 11.318 us, total = 95.878 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1012 total (0 active), Execution time: mean = 567.592 us, total = 574.403 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 490.117 us, total = 8.332 ms, Queueing time: mean = 72.999 us, max = 114.456 us, min = 34.746 us, total = 1.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:33:15,319 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19481 total (0 active) Queueing time: mean = 10.288 us, max = 2.505 ms, min = 563.000 ns, total = 200.420 ms Execution time: mean = 26.971 us, total = 525.415 ms Event stats: event_loop_lag_probe - 19435 total (0 active), Execution time: mean = 26.845 us, total = 521.731 ms, Queueing time: mean = 10.181 us, max = 2.505 ms, min = 1.873 us, total = 197.864 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:34:15,319 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 982 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1024 - cumulative published bytes: 815522 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:34:15,320 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 59326 total (4 active) Queueing time: mean = 75.078 us, max = 225.583 ms, min = -0.023 s, total = 4.454 s Execution time: mean = 230.659 us, total = 13.684 s Event stats: event_loop_lag_probe - 19674 total (0 active), Execution time: mean = 25.878 us, total = 509.123 ms, Queueing time: mean = 10.082 us, max = 6.895 ms, min = 1.268 us, total = 198.355 ms GcsInMemoryStore.Put - 7915 total (0 active), Execution time: mean = 33.740 us, total = 267.053 ms, Queueing time: mean = 243.293 us, max = 225.278 ms, min = 1.495 us, total = 1.926 s RayletLoadPulled - 4919 total (1 active), Execution time: mean = 246.420 us, total = 1.212 s, Queueing time: mean = 75.199 us, max = 1.351 ms, min = 10.621 us, total = 369.903 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4917 total (0 active), Execution time: mean = 75.934 us, total = 373.369 ms, Queueing time: mean = 98.994 us, max = 15.090 ms, min = 11.600 us, total = 486.756 ms NodeManagerService.grpc_client.GetResourceLoad - 4917 total (0 active), Execution time: mean = 1.373 ms, total = 6.751 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 1979 total (0 active), Execution time: mean = 506.412 us, total = 1.002 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1979 total (0 active), Execution time: mean = 40.816 us, total = 80.776 ms, Queueing time: mean = 91.378 us, max = 601.063 us, min = 10.186 us, total = 180.837 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1967 total (0 active), Execution time: mean = 35.375 us, total = 69.583 ms, Queueing time: mean = 107.319 us, max = 1.496 ms, min = 14.438 us, total = 211.097 ms NodeInfoGcsService.grpc_server.CheckAlive - 1967 total (0 active), Execution time: mean = 525.232 us, total = 1.033 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1641 total (1 active), Execution time: mean = 10.542 us, total = 17.300 ms, Queueing time: mean = 73.078 us, max = 715.636 us, min = 7.259 us, total = 119.920 ms HealthCheck - 1638 total (0 active), Execution time: mean = 7.665 us, total = 12.556 ms, Queueing time: mean = 90.643 us, max = 288.247 us, min = 11.335 us, total = 148.473 ms GcsInMemoryStore.Get - 1042 total (0 active), Execution time: mean = 21.349 us, total = 22.246 ms, Queueing time: mean = 12.884 us, max = 137.899 us, min = 2.227 us, total = 13.425 ms InternalKVGcsService.grpc_server.InternalKVGet - 1041 total (0 active), Execution time: mean = 606.457 us, total = 631.321 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1041 total (0 active), Execution time: mean = 56.179 us, total = 58.483 ms, Queueing time: mean = 159.675 us, max = 22.982 ms, min = 9.184 us, total = 166.221 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 982 total (0 active), Execution time: mean = 617.637 us, total = 606.519 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 982 total (0 active), Execution time: mean = 91.516 us, total = 89.868 ms, Queueing time: mean = 107.100 us, max = 235.695 us, min = 15.511 us, total = 105.172 ms GCSServer.deadline_timer.debug_state_dump - 492 total (1 active), Execution time: mean = 1.383 ms, total = 680.280 ms, Queueing time: mean = 22.074 us, max = 166.240 us, min = -0.023 s, total = 10.860 ms GCSServer.deadline_timer.debug_state_event_stats_print - 82 total (1 active, 1 running), Execution time: mean = 1.549 ms, total = 127.010 ms, Queueing time: mean = 56.650 us, max = 137.733 us, min = 14.993 us, total = 4.645 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:34:15,321 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 227174 total (1 active) Queueing time: mean = 42.265 us, max = 58.960 ms, min = 1.891 us, total = 9.602 s Execution time: mean = 221.674 us, total = 50.359 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 103257 total (0 active), Execution time: mean = 459.431 us, total = 47.439 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 103257 total (0 active), Execution time: mean = 23.356 us, total = 2.412 s, Queueing time: mean = 90.587 us, max = 58.960 ms, min = 2.005 us, total = 9.354 s event_loop_lag_probe - 19674 total (0 active), Execution time: mean = 25.301 us, total = 497.771 ms, Queueing time: mean = 8.776 us, max = 395.058 us, min = 1.891 us, total = 172.651 ms GcsTaskManager.GcJobSummary - 985 total (1 active), Execution time: mean = 9.820 us, total = 9.673 ms, Queueing time: mean = 76.195 us, max = 3.642 ms, min = 12.167 us, total = 75.052 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:34:15,321 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24255 total (25 active) Queueing time: mean = 16.111 us, max = 600.706 us, min = 1.622 us, total = 390.777 ms Execution time: mean = 4.691 s, total = 113769.879 s Event stats: event_loop_lag_probe - 19675 total (0 active), Execution time: mean = 25.916 us, total = 509.898 ms, Queueing time: mean = 9.347 us, max = 600.706 us, min = 1.622 us, total = 183.899 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1232 total (0 active), Execution time: mean = 37.749 us, total = 46.506 ms, Queueing time: mean = 87.070 us, max = 408.208 us, min = 5.417 us, total = 107.270 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1232 total (24 active), Execution time: mean = 92.345 s, total = 113768.593 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1024 total (0 active), Execution time: mean = 103.972 us, total = 106.467 ms, Queueing time: mean = 94.116 us, max = 236.271 us, min = 11.318 us, total = 96.375 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1024 total (0 active), Execution time: mean = 564.816 us, total = 578.371 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 490.117 us, total = 8.332 ms, Queueing time: mean = 72.999 us, max = 114.456 us, min = 34.746 us, total = 1.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:34:15,321 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19721 total (0 active) Queueing time: mean = 10.258 us, max = 2.505 ms, min = 563.000 ns, total = 202.290 ms Execution time: mean = 26.902 us, total = 530.536 ms Event stats: event_loop_lag_probe - 19675 total (0 active), Execution time: mean = 26.778 us, total = 526.851 ms, Queueing time: mean = 10.152 us, max = 2.505 ms, min = 1.873 us, total = 199.734 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:35:15,321 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 994 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1036 - cumulative published bytes: 825350 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:35:15,322 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 60045 total (4 active) Queueing time: mean = 74.600 us, max = 225.583 ms, min = -0.023 s, total = 4.479 s Execution time: mean = 230.010 us, total = 13.811 s Event stats: event_loop_lag_probe - 19914 total (0 active), Execution time: mean = 25.880 us, total = 515.365 ms, Queueing time: mean = 10.073 us, max = 6.895 ms, min = 1.268 us, total = 200.599 ms GcsInMemoryStore.Put - 8011 total (0 active), Execution time: mean = 33.388 us, total = 267.470 ms, Queueing time: mean = 240.862 us, max = 225.278 ms, min = 1.495 us, total = 1.930 s RayletLoadPulled - 4979 total (1 active), Execution time: mean = 246.058 us, total = 1.225 s, Queueing time: mean = 75.031 us, max = 1.351 ms, min = 10.621 us, total = 373.578 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4977 total (0 active), Execution time: mean = 75.816 us, total = 377.334 ms, Queueing time: mean = 98.781 us, max = 15.090 ms, min = 11.600 us, total = 491.633 ms NodeManagerService.grpc_client.GetResourceLoad - 4977 total (0 active), Execution time: mean = 1.369 ms, total = 6.812 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2003 total (0 active), Execution time: mean = 504.488 us, total = 1.010 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2003 total (0 active), Execution time: mean = 40.710 us, total = 81.543 ms, Queueing time: mean = 91.203 us, max = 601.063 us, min = 10.186 us, total = 182.681 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1991 total (0 active), Execution time: mean = 35.302 us, total = 70.285 ms, Queueing time: mean = 107.325 us, max = 1.496 ms, min = 14.438 us, total = 213.684 ms NodeInfoGcsService.grpc_server.CheckAlive - 1991 total (0 active), Execution time: mean = 524.074 us, total = 1.043 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1661 total (1 active), Execution time: mean = 10.523 us, total = 17.478 ms, Queueing time: mean = 72.978 us, max = 715.636 us, min = 7.259 us, total = 121.216 ms HealthCheck - 1658 total (0 active), Execution time: mean = 7.669 us, total = 12.714 ms, Queueing time: mean = 90.771 us, max = 288.247 us, min = 11.335 us, total = 150.498 ms GcsInMemoryStore.Get - 1054 total (0 active), Execution time: mean = 21.301 us, total = 22.451 ms, Queueing time: mean = 12.870 us, max = 137.899 us, min = 2.227 us, total = 13.565 ms InternalKVGcsService.grpc_server.InternalKVGet - 1053 total (0 active), Execution time: mean = 604.245 us, total = 636.270 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1053 total (0 active), Execution time: mean = 56.111 us, total = 59.085 ms, Queueing time: mean = 158.812 us, max = 22.982 ms, min = 9.184 us, total = 167.229 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 994 total (0 active), Execution time: mean = 616.334 us, total = 612.636 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 994 total (0 active), Execution time: mean = 91.262 us, total = 90.714 ms, Queueing time: mean = 107.007 us, max = 235.695 us, min = 15.511 us, total = 106.365 ms GCSServer.deadline_timer.debug_state_dump - 498 total (1 active), Execution time: mean = 1.381 ms, total = 687.737 ms, Queueing time: mean = 22.695 us, max = 166.240 us, min = -0.023 s, total = 11.302 ms GCSServer.deadline_timer.debug_state_event_stats_print - 83 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 128.440 ms, Queueing time: mean = 56.593 us, max = 137.733 us, min = 14.993 us, total = 4.697 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:35:15,322 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 229946 total (1 active) Queueing time: mean = 42.057 us, max = 58.960 ms, min = 1.891 us, total = 9.671 s Execution time: mean = 220.861 us, total = 50.786 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 104517 total (0 active), Execution time: mean = 457.719 us, total = 47.839 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 104517 total (0 active), Execution time: mean = 23.279 us, total = 2.433 s, Queueing time: mean = 90.131 us, max = 58.960 ms, min = 2.005 us, total = 9.420 s event_loop_lag_probe - 19914 total (0 active), Execution time: mean = 25.302 us, total = 503.858 ms, Queueing time: mean = 8.776 us, max = 395.058 us, min = 1.891 us, total = 174.763 ms GcsTaskManager.GcJobSummary - 997 total (1 active), Execution time: mean = 9.799 us, total = 9.769 ms, Queueing time: mean = 75.995 us, max = 3.642 ms, min = 12.167 us, total = 75.767 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:35:15,322 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24543 total (25 active) Queueing time: mean = 16.053 us, max = 600.706 us, min = 1.622 us, total = 393.999 ms Execution time: mean = 4.638 s, total = 113829.939 s Event stats: event_loop_lag_probe - 19915 total (0 active), Execution time: mean = 25.877 us, total = 515.338 ms, Queueing time: mean = 9.334 us, max = 600.706 us, min = 1.622 us, total = 185.883 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1244 total (0 active), Execution time: mean = 37.703 us, total = 46.902 ms, Queueing time: mean = 86.664 us, max = 408.208 us, min = 5.417 us, total = 107.810 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1244 total (24 active), Execution time: mean = 91.502 s, total = 113828.641 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1036 total (0 active), Execution time: mean = 103.752 us, total = 107.487 ms, Queueing time: mean = 93.701 us, max = 236.271 us, min = 11.318 us, total = 97.074 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1036 total (0 active), Execution time: mean = 562.960 us, total = 583.227 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 490.117 us, total = 8.332 ms, Queueing time: mean = 72.999 us, max = 114.456 us, min = 34.746 us, total = 1.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:35:15,323 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19961 total (0 active) Queueing time: mean = 10.239 us, max = 2.505 ms, min = 563.000 ns, total = 204.372 ms Execution time: mean = 26.862 us, total = 536.200 ms Event stats: event_loop_lag_probe - 19915 total (0 active), Execution time: mean = 26.739 us, total = 532.516 ms, Queueing time: mean = 10.134 us, max = 2.505 ms, min = 1.873 us, total = 201.816 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:36:15,323 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1006 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1050 - cumulative published bytes: 835246 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:36:15,324 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 60764 total (4 active) Queueing time: mean = 74.186 us, max = 225.583 ms, min = -0.023 s, total = 4.508 s Execution time: mean = 230.093 us, total = 13.981 s Event stats: event_loop_lag_probe - 20154 total (0 active), Execution time: mean = 25.920 us, total = 522.397 ms, Queueing time: mean = 10.078 us, max = 6.895 ms, min = 1.268 us, total = 203.113 ms GcsInMemoryStore.Put - 8107 total (0 active), Execution time: mean = 33.057 us, total = 267.991 ms, Queueing time: mean = 238.650 us, max = 225.278 ms, min = 1.495 us, total = 1.935 s RayletLoadPulled - 5039 total (1 active), Execution time: mean = 246.031 us, total = 1.240 s, Queueing time: mean = 74.908 us, max = 1.351 ms, min = 10.621 us, total = 377.460 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5037 total (0 active), Execution time: mean = 75.811 us, total = 381.861 ms, Queueing time: mean = 98.674 us, max = 15.090 ms, min = 11.600 us, total = 497.021 ms NodeManagerService.grpc_client.GetResourceLoad - 5037 total (0 active), Execution time: mean = 1.370 ms, total = 6.900 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2027 total (0 active), Execution time: mean = 505.179 us, total = 1.024 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2027 total (0 active), Execution time: mean = 40.717 us, total = 82.534 ms, Queueing time: mean = 91.335 us, max = 601.063 us, min = 10.186 us, total = 185.136 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2015 total (0 active), Execution time: mean = 35.313 us, total = 71.155 ms, Queueing time: mean = 107.281 us, max = 1.496 ms, min = 14.438 us, total = 216.172 ms NodeInfoGcsService.grpc_server.CheckAlive - 2015 total (0 active), Execution time: mean = 524.458 us, total = 1.057 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1681 total (1 active), Execution time: mean = 10.529 us, total = 17.699 ms, Queueing time: mean = 73.168 us, max = 715.636 us, min = 7.259 us, total = 122.996 ms HealthCheck - 1678 total (0 active), Execution time: mean = 7.678 us, total = 12.884 ms, Queueing time: mean = 90.594 us, max = 288.247 us, min = 11.335 us, total = 152.016 ms GcsInMemoryStore.Get - 1066 total (0 active), Execution time: mean = 21.277 us, total = 22.681 ms, Queueing time: mean = 12.857 us, max = 137.899 us, min = 2.227 us, total = 13.705 ms InternalKVGcsService.grpc_server.InternalKVGet - 1065 total (0 active), Execution time: mean = 603.918 us, total = 643.173 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1065 total (0 active), Execution time: mean = 56.060 us, total = 59.704 ms, Queueing time: mean = 158.235 us, max = 22.982 ms, min = 9.184 us, total = 168.520 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1006 total (0 active), Execution time: mean = 616.879 us, total = 620.581 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1006 total (0 active), Execution time: mean = 91.239 us, total = 91.787 ms, Queueing time: mean = 107.070 us, max = 235.695 us, min = 15.511 us, total = 107.712 ms GCSServer.deadline_timer.debug_state_dump - 504 total (1 active), Execution time: mean = 1.380 ms, total = 695.478 ms, Queueing time: mean = 23.310 us, max = 166.240 us, min = -0.023 s, total = 11.748 ms GCSServer.deadline_timer.debug_state_event_stats_print - 84 total (1 active, 1 running), Execution time: mean = 1.550 ms, total = 130.180 ms, Queueing time: mean = 56.719 us, max = 137.733 us, min = 14.993 us, total = 4.764 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:36:15,324 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 232718 total (1 active) Queueing time: mean = 42.176 us, max = 58.960 ms, min = 1.891 us, total = 9.815 s Execution time: mean = 221.260 us, total = 51.491 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 105777 total (0 active), Execution time: mean = 458.544 us, total = 48.503 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 105777 total (0 active), Execution time: mean = 23.320 us, total = 2.467 s, Queueing time: mean = 90.388 us, max = 58.960 ms, min = 2.005 us, total = 9.561 s event_loop_lag_probe - 20154 total (0 active), Execution time: mean = 25.367 us, total = 511.245 ms, Queueing time: mean = 8.801 us, max = 395.058 us, min = 1.891 us, total = 177.377 ms GcsTaskManager.GcJobSummary - 1009 total (1 active), Execution time: mean = 9.806 us, total = 9.894 ms, Queueing time: mean = 76.043 us, max = 3.642 ms, min = 12.167 us, total = 76.728 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:36:15,324 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24839 total (25 active) Queueing time: mean = 16.059 us, max = 600.706 us, min = 1.622 us, total = 398.899 ms Execution time: mean = 4.585 s, total = 113890.175 s Event stats: event_loop_lag_probe - 20155 total (0 active), Execution time: mean = 25.884 us, total = 521.689 ms, Queueing time: mean = 9.336 us, max = 600.706 us, min = 1.622 us, total = 188.164 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1258 total (0 active), Execution time: mean = 37.725 us, total = 47.458 ms, Queueing time: mean = 86.659 us, max = 408.208 us, min = 5.417 us, total = 109.018 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1258 total (24 active), Execution time: mean = 90.532 s, total = 113888.860 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1050 total (0 active), Execution time: mean = 104.036 us, total = 109.238 ms, Queueing time: mean = 93.796 us, max = 236.271 us, min = 11.318 us, total = 98.486 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1050 total (0 active), Execution time: mean = 564.058 us, total = 592.261 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 490.117 us, total = 8.332 ms, Queueing time: mean = 72.999 us, max = 114.456 us, min = 34.746 us, total = 1.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:36:15,324 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20201 total (0 active) Queueing time: mean = 10.231 us, max = 2.505 ms, min = 563.000 ns, total = 206.679 ms Execution time: mean = 26.855 us, total = 542.499 ms Event stats: event_loop_lag_probe - 20155 total (0 active), Execution time: mean = 26.734 us, total = 538.815 ms, Queueing time: mean = 10.128 us, max = 2.505 ms, min = 1.873 us, total = 204.123 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:37:15,324 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1018 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1062 - cumulative published bytes: 845074 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:37:15,325 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 61483 total (4 active) Queueing time: mean = 73.781 us, max = 225.583 ms, min = -0.023 s, total = 4.536 s Execution time: mean = 229.752 us, total = 14.126 s Event stats: event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 25.931 us, total = 528.839 ms, Queueing time: mean = 10.077 us, max = 6.895 ms, min = 1.268 us, total = 205.514 ms GcsInMemoryStore.Put - 8203 total (0 active), Execution time: mean = 32.727 us, total = 268.457 ms, Queueing time: mean = 236.312 us, max = 225.278 ms, min = 1.495 us, total = 1.938 s RayletLoadPulled - 5099 total (1 active), Execution time: mean = 246.028 us, total = 1.254 s, Queueing time: mean = 74.852 us, max = 1.351 ms, min = 10.621 us, total = 381.669 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5097 total (0 active), Execution time: mean = 75.788 us, total = 386.290 ms, Queueing time: mean = 98.704 us, max = 15.090 ms, min = 11.600 us, total = 503.095 ms NodeManagerService.grpc_client.GetResourceLoad - 5097 total (0 active), Execution time: mean = 1.368 ms, total = 6.973 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2051 total (0 active), Execution time: mean = 504.238 us, total = 1.034 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2051 total (0 active), Execution time: mean = 40.741 us, total = 83.559 ms, Queueing time: mean = 91.467 us, max = 601.063 us, min = 10.186 us, total = 187.599 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2039 total (0 active), Execution time: mean = 35.294 us, total = 71.965 ms, Queueing time: mean = 107.476 us, max = 1.496 ms, min = 14.438 us, total = 219.144 ms NodeInfoGcsService.grpc_server.CheckAlive - 2039 total (0 active), Execution time: mean = 523.922 us, total = 1.068 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1701 total (1 active), Execution time: mean = 10.508 us, total = 17.874 ms, Queueing time: mean = 73.052 us, max = 715.636 us, min = 7.259 us, total = 124.261 ms HealthCheck - 1698 total (0 active), Execution time: mean = 7.679 us, total = 13.039 ms, Queueing time: mean = 90.777 us, max = 288.247 us, min = 11.335 us, total = 154.140 ms GcsInMemoryStore.Get - 1078 total (0 active), Execution time: mean = 21.252 us, total = 22.910 ms, Queueing time: mean = 12.844 us, max = 137.899 us, min = 2.227 us, total = 13.846 ms InternalKVGcsService.grpc_server.InternalKVGet - 1077 total (0 active), Execution time: mean = 601.916 us, total = 648.264 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1077 total (0 active), Execution time: mean = 56.007 us, total = 60.320 ms, Queueing time: mean = 157.576 us, max = 22.982 ms, min = 9.184 us, total = 169.709 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1018 total (0 active), Execution time: mean = 615.384 us, total = 626.461 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1018 total (0 active), Execution time: mean = 91.063 us, total = 92.702 ms, Queueing time: mean = 107.184 us, max = 235.695 us, min = 15.511 us, total = 109.113 ms GCSServer.deadline_timer.debug_state_dump - 510 total (1 active), Execution time: mean = 1.378 ms, total = 702.895 ms, Queueing time: mean = 23.676 us, max = 166.240 us, min = -0.023 s, total = 12.075 ms GCSServer.deadline_timer.debug_state_event_stats_print - 85 total (1 active, 1 running), Execution time: mean = 1.551 ms, total = 131.826 ms, Queueing time: mean = 57.233 us, max = 137.733 us, min = 14.993 us, total = 4.865 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:37:15,325 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 235490 total (1 active) Queueing time: mean = 42.112 us, max = 58.960 ms, min = 1.891 us, total = 9.917 s Execution time: mean = 220.703 us, total = 51.973 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 107037 total (0 active), Execution time: mean = 457.384 us, total = 48.957 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 107037 total (0 active), Execution time: mean = 23.252 us, total = 2.489 s, Queueing time: mean = 90.250 us, max = 58.960 ms, min = 2.005 us, total = 9.660 s event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 25.371 us, total = 517.417 ms, Queueing time: mean = 8.804 us, max = 395.058 us, min = 1.891 us, total = 179.556 ms GcsTaskManager.GcJobSummary - 1021 total (1 active), Execution time: mean = 9.792 us, total = 9.997 ms, Queueing time: mean = 75.720 us, max = 3.642 ms, min = 12.167 us, total = 77.310 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:37:15,325 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25137 total (25 active) Queueing time: mean = 15.998 us, max = 600.706 us, min = 1.622 us, total = 402.132 ms Execution time: mean = 4.653 s, total = 116950.305 s Event stats: event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 25.844 us, total = 527.061 ms, Queueing time: mean = 9.321 us, max = 600.706 us, min = 1.622 us, total = 190.091 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1275 total (0 active), Execution time: mean = 37.535 us, total = 47.858 ms, Queueing time: mean = 86.010 us, max = 408.208 us, min = 5.417 us, total = 109.663 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1275 total (24 active), Execution time: mean = 91.725 s, total = 116948.979 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1062 total (0 active), Execution time: mean = 103.779 us, total = 110.213 ms, Queueing time: mean = 93.327 us, max = 236.271 us, min = 11.318 us, total = 99.113 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1062 total (0 active), Execution time: mean = 561.943 us, total = 596.784 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 470.971 us, total = 8.477 ms, Queueing time: mean = 70.784 us, max = 114.456 us, min = 33.142 us, total = 1.274 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:37:15,325 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20440 total (0 active) Queueing time: mean = 10.204 us, max = 2.505 ms, min = 563.000 ns, total = 208.571 ms Execution time: mean = 26.786 us, total = 547.500 ms Event stats: event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 26.665 us, total = 543.816 ms, Queueing time: mean = 10.102 us, max = 2.505 ms, min = 1.873 us, total = 206.015 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:38:15,325 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1030 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1074 - cumulative published bytes: 854902 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:38:15,326 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 62202 total (4 active) Queueing time: mean = 73.328 us, max = 225.583 ms, min = -0.023 s, total = 4.561 s Execution time: mean = 229.328 us, total = 14.265 s Event stats: event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 25.924 us, total = 534.914 ms, Queueing time: mean = 10.074 us, max = 6.895 ms, min = 1.268 us, total = 207.857 ms GcsInMemoryStore.Put - 8299 total (0 active), Execution time: mean = 32.403 us, total = 268.910 ms, Queueing time: mean = 234.055 us, max = 225.278 ms, min = 1.495 us, total = 1.942 s RayletLoadPulled - 5159 total (1 active), Execution time: mean = 245.701 us, total = 1.268 s, Queueing time: mean = 74.644 us, max = 1.351 ms, min = 10.621 us, total = 385.087 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5157 total (0 active), Execution time: mean = 75.737 us, total = 390.577 ms, Queueing time: mean = 98.548 us, max = 15.090 ms, min = 11.600 us, total = 508.212 ms NodeManagerService.grpc_client.GetResourceLoad - 5157 total (0 active), Execution time: mean = 1.366 ms, total = 7.043 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2075 total (0 active), Execution time: mean = 502.921 us, total = 1.044 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2075 total (0 active), Execution time: mean = 40.688 us, total = 84.427 ms, Queueing time: mean = 91.267 us, max = 601.063 us, min = 10.186 us, total = 189.380 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2063 total (0 active), Execution time: mean = 35.245 us, total = 72.710 ms, Queueing time: mean = 107.526 us, max = 1.496 ms, min = 14.438 us, total = 221.825 ms NodeInfoGcsService.grpc_server.CheckAlive - 2063 total (0 active), Execution time: mean = 523.202 us, total = 1.079 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1721 total (1 active), Execution time: mean = 10.480 us, total = 18.036 ms, Queueing time: mean = 72.939 us, max = 715.636 us, min = 7.259 us, total = 125.528 ms HealthCheck - 1718 total (0 active), Execution time: mean = 7.674 us, total = 13.183 ms, Queueing time: mean = 90.740 us, max = 288.247 us, min = 11.335 us, total = 155.892 ms GcsInMemoryStore.Get - 1090 total (0 active), Execution time: mean = 21.193 us, total = 23.101 ms, Queueing time: mean = 12.820 us, max = 137.899 us, min = 2.227 us, total = 13.974 ms InternalKVGcsService.grpc_server.InternalKVGet - 1089 total (0 active), Execution time: mean = 600.129 us, total = 653.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1089 total (0 active), Execution time: mean = 55.933 us, total = 60.911 ms, Queueing time: mean = 156.649 us, max = 22.982 ms, min = 9.184 us, total = 170.591 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1030 total (0 active), Execution time: mean = 613.975 us, total = 632.394 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1030 total (0 active), Execution time: mean = 90.976 us, total = 93.705 ms, Queueing time: mean = 106.973 us, max = 235.695 us, min = 15.511 us, total = 110.183 ms GCSServer.deadline_timer.debug_state_dump - 516 total (1 active), Execution time: mean = 1.379 ms, total = 711.386 ms, Queueing time: mean = 24.253 us, max = 166.240 us, min = -0.023 s, total = 12.514 ms GCSServer.deadline_timer.debug_state_event_stats_print - 86 total (1 active, 1 running), Execution time: mean = 1.543 ms, total = 132.685 ms, Queueing time: mean = 56.867 us, max = 137.733 us, min = 14.993 us, total = 4.891 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:38:15,327 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 238262 total (1 active) Queueing time: mean = 41.927 us, max = 58.960 ms, min = 1.891 us, total = 9.990 s Execution time: mean = 219.998 us, total = 52.417 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 108297 total (0 active), Execution time: mean = 455.910 us, total = 49.374 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 108297 total (0 active), Execution time: mean = 23.177 us, total = 2.510 s, Queueing time: mean = 89.845 us, max = 58.960 ms, min = 2.005 us, total = 9.730 s event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 25.359 us, total = 523.249 ms, Queueing time: mean = 8.803 us, max = 395.058 us, min = 1.891 us, total = 181.640 ms GcsTaskManager.GcJobSummary - 1033 total (1 active), Execution time: mean = 9.791 us, total = 10.114 ms, Queueing time: mean = 75.522 us, max = 3.642 ms, min = 12.167 us, total = 78.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:38:15,327 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25425 total (25 active) Queueing time: mean = 15.940 us, max = 600.706 us, min = 1.622 us, total = 405.280 ms Execution time: mean = 4.602 s, total = 117010.514 s Event stats: event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 25.778 us, total = 531.903 ms, Queueing time: mean = 9.297 us, max = 600.706 us, min = 1.622 us, total = 191.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1287 total (0 active), Execution time: mean = 37.456 us, total = 48.206 ms, Queueing time: mean = 85.686 us, max = 408.208 us, min = 5.417 us, total = 110.278 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1287 total (24 active), Execution time: mean = 90.916 s, total = 117009.175 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1074 total (0 active), Execution time: mean = 103.549 us, total = 111.211 ms, Queueing time: mean = 93.024 us, max = 236.271 us, min = 11.318 us, total = 99.908 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1074 total (0 active), Execution time: mean = 561.109 us, total = 602.631 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 470.971 us, total = 8.477 ms, Queueing time: mean = 70.784 us, max = 114.456 us, min = 33.142 us, total = 1.274 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:38:15,327 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20680 total (0 active) Queueing time: mean = 10.180 us, max = 2.505 ms, min = 563.000 ns, total = 210.518 ms Execution time: mean = 26.733 us, total = 552.843 ms Event stats: event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 26.614 us, total = 549.159 ms, Queueing time: mean = 10.079 us, max = 2.505 ms, min = 1.873 us, total = 207.962 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:39:15,327 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1042 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1088 - cumulative published bytes: 864798 - current buffered bytes: 153 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:39:15,328 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 62920 total (4 active) Queueing time: mean = 73.042 us, max = 225.583 ms, min = -0.023 s, total = 4.596 s Execution time: mean = 229.427 us, total = 14.436 s Event stats: event_loop_lag_probe - 20873 total (0 active), Execution time: mean = 25.946 us, total = 541.569 ms, Queueing time: mean = 10.087 us, max = 6.895 ms, min = 1.268 us, total = 210.542 ms GcsInMemoryStore.Put - 8395 total (0 active), Execution time: mean = 32.089 us, total = 269.391 ms, Queueing time: mean = 231.910 us, max = 225.278 ms, min = 1.495 us, total = 1.947 s RayletLoadPulled - 5219 total (1 active), Execution time: mean = 245.773 us, total = 1.283 s, Queueing time: mean = 74.597 us, max = 1.351 ms, min = 10.621 us, total = 389.322 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5217 total (0 active), Execution time: mean = 75.744 us, total = 395.154 ms, Queueing time: mean = 99.689 us, max = 15.090 ms, min = 11.600 us, total = 520.079 ms NodeManagerService.grpc_client.GetResourceLoad - 5217 total (0 active), Execution time: mean = 1.367 ms, total = 7.132 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2099 total (0 active), Execution time: mean = 503.661 us, total = 1.057 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2099 total (0 active), Execution time: mean = 40.677 us, total = 85.380 ms, Queueing time: mean = 91.326 us, max = 601.063 us, min = 10.186 us, total = 191.693 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2087 total (0 active), Execution time: mean = 35.250 us, total = 73.566 ms, Queueing time: mean = 107.478 us, max = 1.496 ms, min = 14.438 us, total = 224.306 ms NodeInfoGcsService.grpc_server.CheckAlive - 2087 total (0 active), Execution time: mean = 523.016 us, total = 1.092 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1741 total (1 active), Execution time: mean = 10.483 us, total = 18.251 ms, Queueing time: mean = 73.069 us, max = 715.636 us, min = 7.259 us, total = 127.214 ms HealthCheck - 1738 total (0 active), Execution time: mean = 7.687 us, total = 13.360 ms, Queueing time: mean = 90.789 us, max = 288.247 us, min = 11.335 us, total = 157.791 ms GcsInMemoryStore.Get - 1102 total (0 active), Execution time: mean = 21.202 us, total = 23.364 ms, Queueing time: mean = 12.819 us, max = 137.899 us, min = 2.227 us, total = 14.127 ms InternalKVGcsService.grpc_server.InternalKVGet - 1101 total (0 active), Execution time: mean = 600.481 us, total = 661.129 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1101 total (0 active), Execution time: mean = 56.103 us, total = 61.770 ms, Queueing time: mean = 155.983 us, max = 22.982 ms, min = 9.184 us, total = 171.737 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1042 total (0 active), Execution time: mean = 614.502 us, total = 640.311 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1042 total (0 active), Execution time: mean = 90.879 us, total = 94.696 ms, Queueing time: mean = 106.988 us, max = 235.695 us, min = 15.511 us, total = 111.482 ms GCSServer.deadline_timer.debug_state_dump - 522 total (1 active), Execution time: mean = 1.377 ms, total = 719.040 ms, Queueing time: mean = 24.726 us, max = 166.240 us, min = -0.023 s, total = 12.907 ms GCSServer.deadline_timer.debug_state_event_stats_print - 87 total (1 active, 1 running), Execution time: mean = 1.546 ms, total = 134.468 ms, Queueing time: mean = 56.929 us, max = 137.733 us, min = 14.993 us, total = 4.953 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:39:15,328 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 241034 total (1 active) Queueing time: mean = 41.981 us, max = 58.960 ms, min = 1.891 us, total = 10.119 s Execution time: mean = 220.079 us, total = 53.046 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 109557 total (0 active), Execution time: mean = 456.075 us, total = 49.966 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 109557 total (0 active), Execution time: mean = 23.187 us, total = 2.540 s, Queueing time: mean = 89.964 us, max = 58.960 ms, min = 2.005 us, total = 9.856 s event_loop_lag_probe - 20874 total (0 active), Execution time: mean = 25.377 us, total = 529.713 ms, Queueing time: mean = 8.810 us, max = 395.058 us, min = 1.891 us, total = 183.894 ms GcsTaskManager.GcJobSummary - 1045 total (1 active), Execution time: mean = 9.776 us, total = 10.216 ms, Queueing time: mean = 75.350 us, max = 3.642 ms, min = 12.167 us, total = 78.741 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:39:15,329 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25721 total (25 active) Queueing time: mean = 15.944 us, max = 600.706 us, min = 1.622 us, total = 410.090 ms Execution time: mean = 4.552 s, total = 117070.598 s Event stats: event_loop_lag_probe - 20874 total (0 active), Execution time: mean = 25.758 us, total = 537.662 ms, Queueing time: mean = 9.288 us, max = 600.706 us, min = 1.622 us, total = 193.882 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1301 total (0 active), Execution time: mean = 37.477 us, total = 48.757 ms, Queueing time: mean = 85.788 us, max = 408.208 us, min = 5.417 us, total = 111.610 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1301 total (24 active), Execution time: mean = 89.984 s, total = 117069.243 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1088 total (0 active), Execution time: mean = 103.429 us, total = 112.530 ms, Queueing time: mean = 93.136 us, max = 236.271 us, min = 11.318 us, total = 101.332 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1088 total (0 active), Execution time: mean = 561.609 us, total = 611.031 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 470.971 us, total = 8.477 ms, Queueing time: mean = 70.784 us, max = 114.456 us, min = 33.142 us, total = 1.274 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:39:15,329 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20920 total (0 active) Queueing time: mean = 10.164 us, max = 2.505 ms, min = 563.000 ns, total = 212.641 ms Execution time: mean = 26.710 us, total = 558.782 ms Event stats: event_loop_lag_probe - 20874 total (0 active), Execution time: mean = 26.593 us, total = 555.098 ms, Queueing time: mean = 10.064 us, max = 2.505 ms, min = 1.873 us, total = 210.085 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:40:15,329 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1054 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1100 - cumulative published bytes: 874626 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:40:15,330 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 63639 total (4 active) Queueing time: mean = 72.666 us, max = 225.583 ms, min = -0.023 s, total = 4.624 s Execution time: mean = 229.250 us, total = 14.589 s Event stats: event_loop_lag_probe - 21113 total (0 active), Execution time: mean = 25.941 us, total = 547.695 ms, Queueing time: mean = 10.078 us, max = 6.895 ms, min = 1.268 us, total = 212.776 ms GcsInMemoryStore.Put - 8491 total (0 active), Execution time: mean = 31.789 us, total = 269.922 ms, Queueing time: mean = 229.777 us, max = 225.278 ms, min = 1.495 us, total = 1.951 s RayletLoadPulled - 5279 total (1 active), Execution time: mean = 245.700 us, total = 1.297 s, Queueing time: mean = 74.595 us, max = 1.351 ms, min = 10.621 us, total = 393.788 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5277 total (0 active), Execution time: mean = 75.745 us, total = 399.707 ms, Queueing time: mean = 99.736 us, max = 15.090 ms, min = 11.600 us, total = 526.306 ms NodeManagerService.grpc_client.GetResourceLoad - 5277 total (0 active), Execution time: mean = 1.366 ms, total = 7.209 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2123 total (0 active), Execution time: mean = 503.098 us, total = 1.068 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2123 total (0 active), Execution time: mean = 40.765 us, total = 86.545 ms, Queueing time: mean = 91.441 us, max = 601.063 us, min = 10.186 us, total = 194.130 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2111 total (0 active), Execution time: mean = 35.304 us, total = 74.527 ms, Queueing time: mean = 107.559 us, max = 1.496 ms, min = 14.438 us, total = 227.057 ms NodeInfoGcsService.grpc_server.CheckAlive - 2111 total (0 active), Execution time: mean = 522.892 us, total = 1.104 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1761 total (1 active), Execution time: mean = 10.478 us, total = 18.452 ms, Queueing time: mean = 73.070 us, max = 715.636 us, min = 7.259 us, total = 128.675 ms HealthCheck - 1758 total (0 active), Execution time: mean = 7.684 us, total = 13.509 ms, Queueing time: mean = 90.650 us, max = 288.247 us, min = 11.335 us, total = 159.363 ms GcsInMemoryStore.Get - 1114 total (0 active), Execution time: mean = 21.196 us, total = 23.612 ms, Queueing time: mean = 12.824 us, max = 137.899 us, min = 2.227 us, total = 14.286 ms InternalKVGcsService.grpc_server.InternalKVGet - 1113 total (0 active), Execution time: mean = 599.714 us, total = 667.481 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1113 total (0 active), Execution time: mean = 56.218 us, total = 62.571 ms, Queueing time: mean = 155.428 us, max = 22.982 ms, min = 9.184 us, total = 172.992 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1054 total (0 active), Execution time: mean = 613.984 us, total = 647.139 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1054 total (0 active), Execution time: mean = 91.034 us, total = 95.950 ms, Queueing time: mean = 107.119 us, max = 235.695 us, min = 15.511 us, total = 112.903 ms GCSServer.deadline_timer.debug_state_dump - 528 total (1 active), Execution time: mean = 1.378 ms, total = 727.374 ms, Queueing time: mean = 25.228 us, max = 166.240 us, min = -0.023 s, total = 13.320 ms GCSServer.deadline_timer.debug_state_event_stats_print - 88 total (1 active, 1 running), Execution time: mean = 1.548 ms, total = 136.194 ms, Queueing time: mean = 57.063 us, max = 137.733 us, min = 14.993 us, total = 5.022 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:40:15,330 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 243806 total (1 active) Queueing time: mean = 41.885 us, max = 58.960 ms, min = 1.891 us, total = 10.212 s Execution time: mean = 219.734 us, total = 53.572 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 110817 total (0 active), Execution time: mean = 455.346 us, total = 50.460 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 110817 total (0 active), Execution time: mean = 23.159 us, total = 2.566 s, Queueing time: mean = 89.756 us, max = 58.960 ms, min = 2.005 us, total = 9.947 s event_loop_lag_probe - 21114 total (0 active), Execution time: mean = 25.363 us, total = 535.513 ms, Queueing time: mean = 8.807 us, max = 395.058 us, min = 1.891 us, total = 185.950 ms GcsTaskManager.GcJobSummary - 1057 total (1 active), Execution time: mean = 9.746 us, total = 10.302 ms, Queueing time: mean = 75.090 us, max = 3.642 ms, min = 12.167 us, total = 79.370 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:40:15,330 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26009 total (25 active) Queueing time: mean = 15.895 us, max = 600.706 us, min = 1.580 us, total = 413.411 ms Execution time: mean = 4.503 s, total = 117130.818 s Event stats: event_loop_lag_probe - 21114 total (0 active), Execution time: mean = 25.693 us, total = 542.479 ms, Queueing time: mean = 9.266 us, max = 600.706 us, min = 1.580 us, total = 195.637 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1313 total (0 active), Execution time: mean = 37.468 us, total = 49.195 ms, Queueing time: mean = 85.543 us, max = 408.208 us, min = 5.417 us, total = 112.318 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1313 total (24 active), Execution time: mean = 89.208 s, total = 117129.451 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1100 total (0 active), Execution time: mean = 103.265 us, total = 113.592 ms, Queueing time: mean = 92.901 us, max = 236.271 us, min = 11.318 us, total = 102.191 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1100 total (0 active), Execution time: mean = 560.703 us, total = 616.774 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 470.971 us, total = 8.477 ms, Queueing time: mean = 70.784 us, max = 114.456 us, min = 33.142 us, total = 1.274 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:40:15,330 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21160 total (0 active) Queueing time: mean = 10.130 us, max = 2.505 ms, min = 563.000 ns, total = 214.344 ms Execution time: mean = 26.628 us, total = 563.438 ms Event stats: event_loop_lag_probe - 21114 total (0 active), Execution time: mean = 26.511 us, total = 559.754 ms, Queueing time: mean = 10.031 us, max = 2.505 ms, min = 1.873 us, total = 211.788 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:41:15,330 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1066 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1111 - cumulative published bytes: 883635 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:41:15,331 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 64358 total (4 active) Queueing time: mean = 72.262 us, max = 225.583 ms, min = -0.023 s, total = 4.651 s Execution time: mean = 228.968 us, total = 14.736 s Event stats: event_loop_lag_probe - 21353 total (0 active), Execution time: mean = 25.934 us, total = 553.779 ms, Queueing time: mean = 10.066 us, max = 6.895 ms, min = 1.268 us, total = 214.950 ms GcsInMemoryStore.Put - 8587 total (0 active), Execution time: mean = 31.485 us, total = 270.362 ms, Queueing time: mean = 227.685 us, max = 225.278 ms, min = 1.495 us, total = 1.955 s RayletLoadPulled - 5339 total (1 active), Execution time: mean = 245.536 us, total = 1.311 s, Queueing time: mean = 74.508 us, max = 1.351 ms, min = 10.621 us, total = 397.801 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5337 total (0 active), Execution time: mean = 75.691 us, total = 403.961 ms, Queueing time: mean = 99.570 us, max = 15.090 ms, min = 11.600 us, total = 531.403 ms NodeManagerService.grpc_client.GetResourceLoad - 5337 total (0 active), Execution time: mean = 1.365 ms, total = 7.284 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2147 total (0 active), Execution time: mean = 502.644 us, total = 1.079 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2147 total (0 active), Execution time: mean = 40.821 us, total = 87.642 ms, Queueing time: mean = 91.439 us, max = 601.063 us, min = 10.186 us, total = 196.319 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2135 total (0 active), Execution time: mean = 35.261 us, total = 75.282 ms, Queueing time: mean = 107.639 us, max = 1.496 ms, min = 14.438 us, total = 229.810 ms NodeInfoGcsService.grpc_server.CheckAlive - 2135 total (0 active), Execution time: mean = 522.437 us, total = 1.115 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1781 total (1 active), Execution time: mean = 10.469 us, total = 18.646 ms, Queueing time: mean = 73.061 us, max = 715.636 us, min = 7.259 us, total = 130.122 ms HealthCheck - 1778 total (0 active), Execution time: mean = 7.679 us, total = 13.654 ms, Queueing time: mean = 90.598 us, max = 288.247 us, min = 11.335 us, total = 161.083 ms GcsInMemoryStore.Get - 1126 total (0 active), Execution time: mean = 21.153 us, total = 23.819 ms, Queueing time: mean = 12.817 us, max = 137.899 us, min = 2.227 us, total = 14.432 ms InternalKVGcsService.grpc_server.InternalKVGet - 1125 total (0 active), Execution time: mean = 598.452 us, total = 673.258 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1125 total (0 active), Execution time: mean = 56.285 us, total = 63.321 ms, Queueing time: mean = 154.780 us, max = 22.982 ms, min = 9.184 us, total = 174.128 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1066 total (0 active), Execution time: mean = 613.336 us, total = 653.816 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1066 total (0 active), Execution time: mean = 90.996 us, total = 97.002 ms, Queueing time: mean = 107.006 us, max = 235.695 us, min = 15.511 us, total = 114.069 ms GCSServer.deadline_timer.debug_state_dump - 534 total (1 active), Execution time: mean = 1.375 ms, total = 734.140 ms, Queueing time: mean = 25.412 us, max = 166.240 us, min = -0.023 s, total = 13.570 ms GCSServer.deadline_timer.debug_state_event_stats_print - 89 total (1 active, 1 running), Execution time: mean = 1.544 ms, total = 137.418 ms, Queueing time: mean = 57.045 us, max = 137.733 us, min = 14.993 us, total = 5.077 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:41:15,331 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 246578 total (1 active) Queueing time: mean = 41.782 us, max = 58.960 ms, min = 1.891 us, total = 10.303 s Execution time: mean = 219.416 us, total = 54.103 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 112077 total (0 active), Execution time: mean = 454.663 us, total = 50.957 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 112077 total (0 active), Execution time: mean = 23.136 us, total = 2.593 s, Queueing time: mean = 89.527 us, max = 58.960 ms, min = 2.005 us, total = 10.034 s event_loop_lag_probe - 21354 total (0 active), Execution time: mean = 25.411 us, total = 542.620 ms, Queueing time: mean = 8.829 us, max = 395.058 us, min = 1.891 us, total = 188.539 ms GcsTaskManager.GcJobSummary - 1069 total (1 active), Execution time: mean = 9.745 us, total = 10.418 ms, Queueing time: mean = 74.946 us, max = 3.642 ms, min = 12.167 us, total = 80.117 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:41:15,332 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26293 total (25 active) Queueing time: mean = 15.868 us, max = 600.706 us, min = 1.580 us, total = 417.216 ms Execution time: mean = 4.457 s, total = 117185.977 s Event stats: event_loop_lag_probe - 21354 total (0 active), Execution time: mean = 25.701 us, total = 548.825 ms, Queueing time: mean = 9.268 us, max = 600.706 us, min = 1.580 us, total = 197.914 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1324 total (0 active), Execution time: mean = 37.487 us, total = 49.633 ms, Queueing time: mean = 85.388 us, max = 408.208 us, min = 5.417 us, total = 113.054 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1324 total (24 active), Execution time: mean = 88.508 s, total = 117184.597 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1111 total (0 active), Execution time: mean = 103.383 us, total = 114.858 ms, Queueing time: mean = 92.694 us, max = 236.271 us, min = 11.318 us, total = 102.983 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1111 total (0 active), Execution time: mean = 560.133 us, total = 622.308 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 470.971 us, total = 8.477 ms, Queueing time: mean = 70.784 us, max = 114.456 us, min = 33.142 us, total = 1.274 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:41:15,332 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21400 total (0 active) Queueing time: mean = 10.110 us, max = 2.505 ms, min = 563.000 ns, total = 216.346 ms Execution time: mean = 26.574 us, total = 568.690 ms Event stats: event_loop_lag_probe - 21354 total (0 active), Execution time: mean = 26.459 us, total = 565.006 ms, Queueing time: mean = 10.012 us, max = 2.505 ms, min = 1.873 us, total = 213.790 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:42:15,332 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1077 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1123 - cumulative published bytes: 893463 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:42:15,333 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 65066 total (4 active) Queueing time: mean = 71.855 us, max = 225.583 ms, min = -0.023 s, total = 4.675 s Execution time: mean = 228.210 us, total = 14.849 s Event stats: event_loop_lag_probe - 21593 total (0 active), Execution time: mean = 25.905 us, total = 559.367 ms, Queueing time: mean = 10.052 us, max = 6.895 ms, min = 1.268 us, total = 217.044 ms GcsInMemoryStore.Put - 8681 total (0 active), Execution time: mean = 31.191 us, total = 270.772 ms, Queueing time: mean = 225.630 us, max = 225.278 ms, min = 1.495 us, total = 1.959 s RayletLoadPulled - 5399 total (1 active), Execution time: mean = 245.193 us, total = 1.324 s, Queueing time: mean = 74.469 us, max = 1.351 ms, min = 10.621 us, total = 402.056 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5397 total (0 active), Execution time: mean = 75.588 us, total = 407.946 ms, Queueing time: mean = 99.432 us, max = 15.090 ms, min = 11.600 us, total = 536.635 ms NodeManagerService.grpc_client.GetResourceLoad - 5397 total (0 active), Execution time: mean = 1.359 ms, total = 7.336 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2169 total (0 active), Execution time: mean = 500.995 us, total = 1.087 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2169 total (0 active), Execution time: mean = 40.726 us, total = 88.335 ms, Queueing time: mean = 91.359 us, max = 601.063 us, min = 10.186 us, total = 198.158 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2159 total (0 active), Execution time: mean = 35.202 us, total = 76.000 ms, Queueing time: mean = 107.436 us, max = 1.496 ms, min = 14.438 us, total = 231.953 ms NodeInfoGcsService.grpc_server.CheckAlive - 2159 total (0 active), Execution time: mean = 520.430 us, total = 1.124 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1801 total (1 active), Execution time: mean = 10.465 us, total = 18.848 ms, Queueing time: mean = 73.045 us, max = 715.636 us, min = 7.259 us, total = 131.554 ms HealthCheck - 1798 total (0 active), Execution time: mean = 7.670 us, total = 13.791 ms, Queueing time: mean = 90.595 us, max = 288.247 us, min = 11.335 us, total = 162.891 ms GcsInMemoryStore.Get - 1137 total (0 active), Execution time: mean = 21.225 us, total = 24.133 ms, Queueing time: mean = 12.797 us, max = 137.899 us, min = 2.227 us, total = 14.551 ms InternalKVGcsService.grpc_server.InternalKVGet - 1136 total (0 active), Execution time: mean = 596.586 us, total = 677.721 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1136 total (0 active), Execution time: mean = 56.204 us, total = 63.848 ms, Queueing time: mean = 154.039 us, max = 22.982 ms, min = 9.184 us, total = 174.988 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1077 total (0 active), Execution time: mean = 611.615 us, total = 658.709 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1077 total (0 active), Execution time: mean = 90.818 us, total = 97.811 ms, Queueing time: mean = 106.831 us, max = 235.695 us, min = 15.511 us, total = 115.057 ms GCSServer.deadline_timer.debug_state_dump - 540 total (1 active), Execution time: mean = 1.374 ms, total = 741.982 ms, Queueing time: mean = 25.677 us, max = 166.240 us, min = -0.023 s, total = 13.865 ms GCSServer.deadline_timer.debug_state_event_stats_print - 90 total (1 active, 1 running), Execution time: mean = 1.545 ms, total = 139.050 ms, Queueing time: mean = 56.750 us, max = 137.733 us, min = 14.993 us, total = 5.108 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:42:15,333 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 249349 total (1 active) Queueing time: mean = 41.477 us, max = 58.960 ms, min = 1.891 us, total = 10.342 s Execution time: mean = 218.248 us, total = 54.420 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 113337 total (0 active), Execution time: mean = 452.199 us, total = 51.251 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 113337 total (0 active), Execution time: mean = 23.028 us, total = 2.610 s, Queueing time: mean = 88.857 us, max = 58.960 ms, min = 2.005 us, total = 10.071 s event_loop_lag_probe - 21593 total (0 active), Execution time: mean = 25.403 us, total = 548.523 ms, Queueing time: mean = 8.829 us, max = 395.058 us, min = 1.891 us, total = 190.638 ms GcsTaskManager.GcJobSummary - 1081 total (1 active), Execution time: mean = 9.733 us, total = 10.521 ms, Queueing time: mean = 74.770 us, max = 3.642 ms, min = 12.167 us, total = 80.826 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:42:15,333 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26618 total (25 active) Queueing time: mean = 15.781 us, max = 600.706 us, min = 1.580 us, total = 420.061 ms Execution time: mean = 4.811 s, total = 128046.074 s Event stats: event_loop_lag_probe - 21594 total (0 active), Execution time: mean = 25.669 us, total = 554.299 ms, Queueing time: mean = 9.255 us, max = 600.706 us, min = 1.580 us, total = 199.850 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1354 total (0 active), Execution time: mean = 36.978 us, total = 50.068 ms, Queueing time: mean = 83.859 us, max = 408.208 us, min = 5.163 us, total = 113.545 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1354 total (24 active), Execution time: mean = 94.568 s, total = 128044.683 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1123 total (0 active), Execution time: mean = 103.051 us, total = 115.726 ms, Queueing time: mean = 92.042 us, max = 236.271 us, min = 11.318 us, total = 103.364 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1123 total (0 active), Execution time: mean = 557.350 us, total = 625.904 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 459.770 us, total = 8.736 ms, Queueing time: mean = 68.977 us, max = 114.456 us, min = 33.142 us, total = 1.311 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:42:15,333 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21640 total (0 active) Queueing time: mean = 10.084 us, max = 2.505 ms, min = 563.000 ns, total = 218.209 ms Execution time: mean = 26.513 us, total = 573.747 ms Event stats: event_loop_lag_probe - 21594 total (0 active), Execution time: mean = 26.399 us, total = 570.063 ms, Queueing time: mean = 9.987 us, max = 2.505 ms, min = 1.873 us, total = 215.653 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:43:15,333 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1089 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1136 - cumulative published bytes: 903325 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:43:15,334 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 65785 total (4 active) Queueing time: mean = 71.314 us, max = 225.583 ms, min = -0.023 s, total = 4.691 s Execution time: mean = 227.340 us, total = 14.956 s Event stats: event_loop_lag_probe - 21833 total (0 active), Execution time: mean = 25.867 us, total = 564.756 ms, Queueing time: mean = 10.033 us, max = 6.895 ms, min = 1.268 us, total = 219.053 ms GcsInMemoryStore.Put - 8777 total (0 active), Execution time: mean = 30.885 us, total = 271.081 ms, Queueing time: mean = 223.565 us, max = 225.278 ms, min = 1.495 us, total = 1.962 s RayletLoadPulled - 5459 total (1 active), Execution time: mean = 244.721 us, total = 1.336 s, Queueing time: mean = 74.193 us, max = 1.351 ms, min = 10.621 us, total = 405.021 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5457 total (0 active), Execution time: mean = 75.400 us, total = 411.456 ms, Queueing time: mean = 98.734 us, max = 15.090 ms, min = 11.600 us, total = 538.791 ms NodeManagerService.grpc_client.GetResourceLoad - 5457 total (0 active), Execution time: mean = 1.354 ms, total = 7.386 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2193 total (0 active), Execution time: mean = 498.514 us, total = 1.093 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2193 total (0 active), Execution time: mean = 40.500 us, total = 88.815 ms, Queueing time: mean = 90.747 us, max = 601.063 us, min = 10.186 us, total = 199.008 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2183 total (0 active), Execution time: mean = 35.109 us, total = 76.643 ms, Queueing time: mean = 106.882 us, max = 1.496 ms, min = 14.438 us, total = 233.324 ms NodeInfoGcsService.grpc_server.CheckAlive - 2183 total (0 active), Execution time: mean = 518.200 us, total = 1.131 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1821 total (1 active), Execution time: mean = 10.446 us, total = 19.023 ms, Queueing time: mean = 72.830 us, max = 715.636 us, min = 7.259 us, total = 132.624 ms HealthCheck - 1818 total (0 active), Execution time: mean = 7.648 us, total = 13.905 ms, Queueing time: mean = 90.091 us, max = 288.247 us, min = 11.335 us, total = 163.785 ms GcsInMemoryStore.Get - 1149 total (0 active), Execution time: mean = 21.132 us, total = 24.281 ms, Queueing time: mean = 12.742 us, max = 137.899 us, min = 2.227 us, total = 14.641 ms InternalKVGcsService.grpc_server.InternalKVGet - 1148 total (0 active), Execution time: mean = 593.647 us, total = 681.506 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1148 total (0 active), Execution time: mean = 56.018 us, total = 64.308 ms, Queueing time: mean = 152.678 us, max = 22.982 ms, min = 9.184 us, total = 175.274 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1089 total (0 active), Execution time: mean = 609.070 us, total = 663.277 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1089 total (0 active), Execution time: mean = 90.510 us, total = 98.565 ms, Queueing time: mean = 106.299 us, max = 235.695 us, min = 15.511 us, total = 115.760 ms GCSServer.deadline_timer.debug_state_dump - 546 total (1 active), Execution time: mean = 1.373 ms, total = 749.929 ms, Queueing time: mean = 25.670 us, max = 166.240 us, min = -0.023 s, total = 14.016 ms GCSServer.deadline_timer.debug_state_event_stats_print - 91 total (1 active, 1 running), Execution time: mean = 1.545 ms, total = 140.628 ms, Queueing time: mean = 56.379 us, max = 137.733 us, min = 14.993 us, total = 5.131 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:43:15,334 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 252121 total (1 active) Queueing time: mean = 41.144 us, max = 58.960 ms, min = 1.891 us, total = 10.373 s Execution time: mean = 217.019 us, total = 54.715 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 114597 total (0 active), Execution time: mean = 449.621 us, total = 51.525 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 114597 total (0 active), Execution time: mean = 22.909 us, total = 2.625 s, Queueing time: mean = 88.129 us, max = 58.960 ms, min = 2.005 us, total = 10.099 s event_loop_lag_probe - 21833 total (0 active), Execution time: mean = 25.378 us, total = 554.071 ms, Queueing time: mean = 8.821 us, max = 395.058 us, min = 1.891 us, total = 192.590 ms GcsTaskManager.GcJobSummary - 1093 total (1 active), Execution time: mean = 9.715 us, total = 10.618 ms, Queueing time: mean = 74.426 us, max = 3.642 ms, min = 12.167 us, total = 81.348 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:43:15,334 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26910 total (25 active) Queueing time: mean = 15.708 us, max = 600.706 us, min = 1.580 us, total = 422.698 ms Execution time: mean = 4.761 s, total = 128106.223 s Event stats: event_loop_lag_probe - 21834 total (0 active), Execution time: mean = 25.631 us, total = 559.635 ms, Queueing time: mean = 9.240 us, max = 600.706 us, min = 1.580 us, total = 201.755 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1367 total (0 active), Execution time: mean = 36.862 us, total = 50.391 ms, Queueing time: mean = 83.238 us, max = 408.208 us, min = 5.163 us, total = 113.786 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1367 total (24 active), Execution time: mean = 93.712 s, total = 128104.821 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1136 total (0 active), Execution time: mean = 102.790 us, total = 116.769 ms, Queueing time: mean = 91.423 us, max = 236.271 us, min = 11.318 us, total = 103.856 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1136 total (0 active), Execution time: mean = 554.740 us, total = 630.185 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 459.770 us, total = 8.736 ms, Queueing time: mean = 68.977 us, max = 114.456 us, min = 33.142 us, total = 1.311 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:43:15,334 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21880 total (0 active) Queueing time: mean = 10.058 us, max = 2.505 ms, min = 563.000 ns, total = 220.070 ms Execution time: mean = 26.444 us, total = 578.588 ms Event stats: event_loop_lag_probe - 21834 total (0 active), Execution time: mean = 26.331 us, total = 574.903 ms, Queueing time: mean = 9.962 us, max = 2.505 ms, min = 1.873 us, total = 217.514 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:44:15,334 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1101 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1148 - cumulative published bytes: 913153 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:44:15,335 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 66504 total (4 active) Queueing time: mean = 70.849 us, max = 225.583 ms, min = -0.023 s, total = 4.712 s Execution time: mean = 226.679 us, total = 15.075 s Event stats: event_loop_lag_probe - 22073 total (0 active), Execution time: mean = 25.828 us, total = 570.105 ms, Queueing time: mean = 10.015 us, max = 6.895 ms, min = 1.268 us, total = 221.063 ms GcsInMemoryStore.Put - 8873 total (0 active), Execution time: mean = 30.588 us, total = 271.412 ms, Queueing time: mean = 221.587 us, max = 225.278 ms, min = 1.495 us, total = 1.966 s RayletLoadPulled - 5519 total (1 active), Execution time: mean = 244.252 us, total = 1.348 s, Queueing time: mean = 74.062 us, max = 1.351 ms, min = 10.621 us, total = 408.748 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5517 total (0 active), Execution time: mean = 75.239 us, total = 415.092 ms, Queueing time: mean = 98.324 us, max = 15.090 ms, min = 11.600 us, total = 542.456 ms NodeManagerService.grpc_client.GetResourceLoad - 5517 total (0 active), Execution time: mean = 1.350 ms, total = 7.447 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2217 total (0 active), Execution time: mean = 496.401 us, total = 1.101 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2217 total (0 active), Execution time: mean = 40.299 us, total = 89.342 ms, Queueing time: mean = 90.228 us, max = 601.063 us, min = 10.186 us, total = 200.035 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2207 total (0 active), Execution time: mean = 35.025 us, total = 77.299 ms, Queueing time: mean = 106.617 us, max = 1.496 ms, min = 14.438 us, total = 235.303 ms NodeInfoGcsService.grpc_server.CheckAlive - 2207 total (0 active), Execution time: mean = 516.969 us, total = 1.141 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1841 total (1 active), Execution time: mean = 10.435 us, total = 19.211 ms, Queueing time: mean = 72.680 us, max = 715.636 us, min = 7.259 us, total = 133.803 ms HealthCheck - 1838 total (0 active), Execution time: mean = 7.630 us, total = 14.023 ms, Queueing time: mean = 89.717 us, max = 288.247 us, min = 11.335 us, total = 164.899 ms GcsInMemoryStore.Get - 1161 total (0 active), Execution time: mean = 21.053 us, total = 24.443 ms, Queueing time: mean = 12.708 us, max = 137.899 us, min = 2.227 us, total = 14.754 ms InternalKVGcsService.grpc_server.InternalKVGet - 1160 total (0 active), Execution time: mean = 591.143 us, total = 685.726 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1160 total (0 active), Execution time: mean = 55.877 us, total = 64.817 ms, Queueing time: mean = 151.563 us, max = 22.982 ms, min = 9.184 us, total = 175.813 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1101 total (0 active), Execution time: mean = 607.063 us, total = 668.377 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1101 total (0 active), Execution time: mean = 90.231 us, total = 99.345 ms, Queueing time: mean = 105.867 us, max = 235.695 us, min = 15.511 us, total = 116.559 ms GCSServer.deadline_timer.debug_state_dump - 552 total (1 active), Execution time: mean = 1.372 ms, total = 757.273 ms, Queueing time: mean = 25.780 us, max = 166.240 us, min = -0.023 s, total = 14.230 ms GCSServer.deadline_timer.debug_state_event_stats_print - 92 total (1 active, 1 running), Execution time: mean = 1.538 ms, total = 141.529 ms, Queueing time: mean = 55.918 us, max = 137.733 us, min = 13.943 us, total = 5.144 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:44:15,336 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 254893 total (1 active) Queueing time: mean = 40.946 us, max = 58.960 ms, min = 1.891 us, total = 10.437 s Execution time: mean = 216.247 us, total = 55.120 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 115857 total (0 active), Execution time: mean = 447.999 us, total = 51.904 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 115857 total (0 active), Execution time: mean = 22.832 us, total = 2.645 s, Queueing time: mean = 87.695 us, max = 58.960 ms, min = 2.005 us, total = 10.160 s event_loop_lag_probe - 22073 total (0 active), Execution time: mean = 25.375 us, total = 560.103 ms, Queueing time: mean = 8.822 us, max = 395.058 us, min = 1.891 us, total = 194.730 ms GcsTaskManager.GcJobSummary - 1105 total (1 active), Execution time: mean = 9.698 us, total = 10.716 ms, Queueing time: mean = 74.160 us, max = 3.642 ms, min = 12.167 us, total = 81.947 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:44:15,336 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27198 total (25 active) Queueing time: mean = 15.651 us, max = 600.706 us, min = 1.580 us, total = 425.664 ms Execution time: mean = 4.712 s, total = 128166.341 s Event stats: event_loop_lag_probe - 22074 total (0 active), Execution time: mean = 25.589 us, total = 564.848 ms, Queueing time: mean = 9.225 us, max = 600.706 us, min = 1.580 us, total = 203.643 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1379 total (0 active), Execution time: mean = 36.766 us, total = 50.700 ms, Queueing time: mean = 82.841 us, max = 408.208 us, min = 5.163 us, total = 114.238 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1379 total (24 active), Execution time: mean = 92.940 s, total = 128164.929 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1148 total (0 active), Execution time: mean = 102.536 us, total = 117.711 ms, Queueing time: mean = 91.011 us, max = 236.271 us, min = 11.318 us, total = 104.481 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1148 total (0 active), Execution time: mean = 552.606 us, total = 634.391 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 459.770 us, total = 8.736 ms, Queueing time: mean = 68.977 us, max = 114.456 us, min = 33.142 us, total = 1.311 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:44:15,336 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22120 total (0 active) Queueing time: mean = 10.039 us, max = 2.505 ms, min = 563.000 ns, total = 222.056 ms Execution time: mean = 26.395 us, total = 583.860 ms Event stats: event_loop_lag_probe - 22074 total (0 active), Execution time: mean = 26.283 us, total = 580.176 ms, Queueing time: mean = 9.944 us, max = 2.505 ms, min = 1.873 us, total = 219.500 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:45:15,336 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1113 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1160 - cumulative published bytes: 922981 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:45:15,337 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 67223 total (4 active) Queueing time: mean = 70.522 us, max = 225.583 ms, min = -0.023 s, total = 4.741 s Execution time: mean = 226.618 us, total = 15.234 s Event stats: event_loop_lag_probe - 22313 total (0 active), Execution time: mean = 25.844 us, total = 576.650 ms, Queueing time: mean = 10.011 us, max = 6.895 ms, min = 1.268 us, total = 223.365 ms GcsInMemoryStore.Put - 8969 total (0 active), Execution time: mean = 30.320 us, total = 271.936 ms, Queueing time: mean = 219.688 us, max = 225.278 ms, min = 1.495 us, total = 1.970 s RayletLoadPulled - 5579 total (1 active), Execution time: mean = 244.176 us, total = 1.362 s, Queueing time: mean = 74.118 us, max = 1.351 ms, min = 10.621 us, total = 413.502 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5577 total (0 active), Execution time: mean = 75.243 us, total = 419.628 ms, Queueing time: mean = 98.450 us, max = 15.090 ms, min = 11.600 us, total = 549.055 ms NodeManagerService.grpc_client.GetResourceLoad - 5577 total (0 active), Execution time: mean = 1.350 ms, total = 7.529 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2241 total (0 active), Execution time: mean = 496.223 us, total = 1.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2241 total (0 active), Execution time: mean = 40.230 us, total = 90.155 ms, Queueing time: mean = 90.117 us, max = 601.063 us, min = 10.186 us, total = 201.951 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2231 total (0 active), Execution time: mean = 35.081 us, total = 78.265 ms, Queueing time: mean = 106.656 us, max = 1.496 ms, min = 14.438 us, total = 237.949 ms NodeInfoGcsService.grpc_server.CheckAlive - 2231 total (0 active), Execution time: mean = 517.110 us, total = 1.154 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1861 total (1 active), Execution time: mean = 10.432 us, total = 19.414 ms, Queueing time: mean = 72.686 us, max = 715.636 us, min = 7.259 us, total = 135.268 ms HealthCheck - 1858 total (0 active), Execution time: mean = 7.633 us, total = 14.181 ms, Queueing time: mean = 89.901 us, max = 288.247 us, min = 11.335 us, total = 167.037 ms GcsInMemoryStore.Get - 1173 total (0 active), Execution time: mean = 21.028 us, total = 24.666 ms, Queueing time: mean = 12.692 us, max = 137.899 us, min = 2.227 us, total = 14.888 ms InternalKVGcsService.grpc_server.InternalKVGet - 1172 total (0 active), Execution time: mean = 590.316 us, total = 691.851 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1172 total (0 active), Execution time: mean = 55.820 us, total = 65.421 ms, Queueing time: mean = 150.865 us, max = 22.982 ms, min = 9.184 us, total = 176.814 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1113 total (0 active), Execution time: mean = 606.772 us, total = 675.338 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1113 total (0 active), Execution time: mean = 90.286 us, total = 100.489 ms, Queueing time: mean = 105.936 us, max = 235.695 us, min = 15.511 us, total = 117.907 ms GCSServer.deadline_timer.debug_state_dump - 558 total (1 active), Execution time: mean = 1.371 ms, total = 764.917 ms, Queueing time: mean = 26.162 us, max = 166.240 us, min = -0.023 s, total = 14.598 ms GCSServer.deadline_timer.debug_state_event_stats_print - 93 total (1 active, 1 running), Execution time: mean = 1.538 ms, total = 143.075 ms, Queueing time: mean = 56.092 us, max = 137.733 us, min = 13.943 us, total = 5.217 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:45:15,337 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 257665 total (1 active) Queueing time: mean = 40.940 us, max = 58.960 ms, min = 1.891 us, total = 10.549 s Execution time: mean = 216.257 us, total = 55.722 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 117117 total (0 active), Execution time: mean = 448.014 us, total = 52.470 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 117117 total (0 active), Execution time: mean = 22.833 us, total = 2.674 s, Queueing time: mean = 87.680 us, max = 58.960 ms, min = 2.005 us, total = 10.269 s event_loop_lag_probe - 22313 total (0 active), Execution time: mean = 25.403 us, total = 566.828 ms, Queueing time: mean = 8.834 us, max = 395.058 us, min = 1.891 us, total = 197.112 ms GcsTaskManager.GcJobSummary - 1117 total (1 active), Execution time: mean = 9.697 us, total = 10.831 ms, Queueing time: mean = 74.244 us, max = 3.642 ms, min = 12.167 us, total = 82.931 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:45:15,337 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27486 total (25 active) Queueing time: mean = 15.647 us, max = 600.706 us, min = 1.580 us, total = 430.072 ms Execution time: mean = 4.665 s, total = 128226.533 s Event stats: event_loop_lag_probe - 22314 total (0 active), Execution time: mean = 25.568 us, total = 570.524 ms, Queueing time: mean = 9.219 us, max = 600.706 us, min = 1.580 us, total = 205.713 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1391 total (0 active), Execution time: mean = 36.768 us, total = 51.145 ms, Queueing time: mean = 82.987 us, max = 408.208 us, min = 5.163 us, total = 115.435 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1391 total (24 active), Execution time: mean = 92.182 s, total = 128225.107 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1160 total (0 active), Execution time: mean = 102.445 us, total = 118.836 ms, Queueing time: mean = 91.053 us, max = 236.271 us, min = 11.318 us, total = 105.622 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1160 total (0 active), Execution time: mean = 552.669 us, total = 641.096 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 459.770 us, total = 8.736 ms, Queueing time: mean = 68.977 us, max = 114.456 us, min = 33.142 us, total = 1.311 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:45:15,338 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22360 total (0 active) Queueing time: mean = 10.028 us, max = 2.505 ms, min = 563.000 ns, total = 224.237 ms Execution time: mean = 26.371 us, total = 589.664 ms Event stats: event_loop_lag_probe - 22314 total (0 active), Execution time: mean = 26.261 us, total = 585.980 ms, Queueing time: mean = 9.935 us, max = 2.505 ms, min = 1.873 us, total = 221.680 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:46:15,338 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1125 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1174 - cumulative published bytes: 932877 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:46:15,339 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 67942 total (4 active) Queueing time: mean = 70.227 us, max = 225.583 ms, min = -0.023 s, total = 4.771 s Execution time: mean = 226.690 us, total = 15.402 s Event stats: event_loop_lag_probe - 22553 total (0 active), Execution time: mean = 25.860 us, total = 583.228 ms, Queueing time: mean = 10.008 us, max = 6.895 ms, min = 1.268 us, total = 225.720 ms GcsInMemoryStore.Put - 9065 total (0 active), Execution time: mean = 30.053 us, total = 272.433 ms, Queueing time: mean = 218.088 us, max = 225.278 ms, min = 1.495 us, total = 1.977 s RayletLoadPulled - 5639 total (1 active), Execution time: mean = 244.365 us, total = 1.378 s, Queueing time: mean = 74.244 us, max = 1.351 ms, min = 10.621 us, total = 418.665 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5637 total (0 active), Execution time: mean = 75.252 us, total = 424.197 ms, Queueing time: mean = 98.398 us, max = 15.090 ms, min = 11.600 us, total = 554.672 ms NodeManagerService.grpc_client.GetResourceLoad - 5637 total (0 active), Execution time: mean = 1.352 ms, total = 7.620 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2265 total (0 active), Execution time: mean = 495.788 us, total = 1.123 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2265 total (0 active), Execution time: mean = 40.166 us, total = 90.975 ms, Queueing time: mean = 90.106 us, max = 601.063 us, min = 10.186 us, total = 204.090 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2255 total (0 active), Execution time: mean = 35.153 us, total = 79.271 ms, Queueing time: mean = 106.655 us, max = 1.496 ms, min = 14.438 us, total = 240.507 ms NodeInfoGcsService.grpc_server.CheckAlive - 2255 total (0 active), Execution time: mean = 517.196 us, total = 1.166 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1881 total (1 active), Execution time: mean = 10.434 us, total = 19.627 ms, Queueing time: mean = 72.709 us, max = 715.636 us, min = 7.259 us, total = 136.765 ms HealthCheck - 1878 total (0 active), Execution time: mean = 7.634 us, total = 14.337 ms, Queueing time: mean = 89.966 us, max = 288.247 us, min = 11.335 us, total = 168.956 ms GcsInMemoryStore.Get - 1185 total (0 active), Execution time: mean = 20.995 us, total = 24.879 ms, Queueing time: mean = 12.674 us, max = 137.899 us, min = 2.227 us, total = 15.018 ms InternalKVGcsService.grpc_server.InternalKVGet - 1184 total (0 active), Execution time: mean = 589.411 us, total = 697.862 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1184 total (0 active), Execution time: mean = 55.793 us, total = 66.059 ms, Queueing time: mean = 150.307 us, max = 22.982 ms, min = 9.184 us, total = 177.963 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1125 total (0 active), Execution time: mean = 606.338 us, total = 682.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1125 total (0 active), Execution time: mean = 90.336 us, total = 101.628 ms, Queueing time: mean = 105.817 us, max = 235.695 us, min = 15.511 us, total = 119.044 ms GCSServer.deadline_timer.debug_state_dump - 564 total (1 active), Execution time: mean = 1.370 ms, total = 772.547 ms, Queueing time: mean = 26.557 us, max = 166.240 us, min = -0.023 s, total = 14.978 ms GCSServer.deadline_timer.debug_state_event_stats_print - 94 total (1 active, 1 running), Execution time: mean = 1.539 ms, total = 144.652 ms, Queueing time: mean = 56.118 us, max = 137.733 us, min = 13.943 us, total = 5.275 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:46:15,339 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 260437 total (1 active) Queueing time: mean = 40.957 us, max = 58.960 ms, min = 1.891 us, total = 10.667 s Execution time: mean = 216.277 us, total = 56.327 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 118377 total (0 active), Execution time: mean = 448.036 us, total = 53.037 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 118377 total (0 active), Execution time: mean = 22.846 us, total = 2.704 s, Queueing time: mean = 87.713 us, max = 58.960 ms, min = 2.005 us, total = 10.383 s event_loop_lag_probe - 22553 total (0 active), Execution time: mean = 25.455 us, total = 574.086 ms, Queueing time: mean = 8.855 us, max = 395.058 us, min = 1.891 us, total = 199.709 ms GcsTaskManager.GcJobSummary - 1129 total (1 active), Execution time: mean = 9.699 us, total = 10.950 ms, Queueing time: mean = 74.212 us, max = 3.642 ms, min = 12.167 us, total = 83.786 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:46:15,339 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27782 total (25 active) Queueing time: mean = 15.654 us, max = 600.706 us, min = 1.580 us, total = 434.911 ms Execution time: mean = 4.618 s, total = 128286.723 s Event stats: event_loop_lag_probe - 22554 total (0 active), Execution time: mean = 25.535 us, total = 575.927 ms, Queueing time: mean = 9.213 us, max = 600.706 us, min = 1.580 us, total = 207.799 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1405 total (0 active), Execution time: mean = 36.765 us, total = 51.655 ms, Queueing time: mean = 83.088 us, max = 408.208 us, min = 5.163 us, total = 116.739 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1405 total (24 active), Execution time: mean = 91.306 s, total = 128285.281 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1174 total (0 active), Execution time: mean = 102.414 us, total = 120.234 ms, Queueing time: mean = 91.203 us, max = 236.271 us, min = 11.318 us, total = 107.072 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1174 total (0 active), Execution time: mean = 553.062 us, total = 649.295 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 459.770 us, total = 8.736 ms, Queueing time: mean = 68.977 us, max = 114.456 us, min = 33.142 us, total = 1.311 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:46:15,339 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22600 total (0 active) Queueing time: mean = 10.019 us, max = 2.505 ms, min = 563.000 ns, total = 226.432 ms Execution time: mean = 26.354 us, total = 595.603 ms Event stats: event_loop_lag_probe - 22554 total (0 active), Execution time: mean = 26.245 us, total = 591.919 ms, Queueing time: mean = 9.926 us, max = 2.505 ms, min = 1.873 us, total = 223.876 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:47:15,340 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1137 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1186 - cumulative published bytes: 942705 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:47:15,340 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 68660 total (4 active) Queueing time: mean = 69.939 us, max = 225.583 ms, min = -0.023 s, total = 4.802 s Execution time: mean = 227.116 us, total = 15.594 s Event stats: event_loop_lag_probe - 22792 total (0 active), Execution time: mean = 25.879 us, total = 589.829 ms, Queueing time: mean = 10.009 us, max = 6.895 ms, min = 1.268 us, total = 228.133 ms GcsInMemoryStore.Put - 9161 total (0 active), Execution time: mean = 29.796 us, total = 272.961 ms, Queueing time: mean = 216.280 us, max = 225.278 ms, min = 1.495 us, total = 1.981 s RayletLoadPulled - 5699 total (1 active), Execution time: mean = 244.373 us, total = 1.393 s, Queueing time: mean = 74.267 us, max = 1.351 ms, min = 10.621 us, total = 423.250 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5697 total (0 active), Execution time: mean = 75.264 us, total = 428.778 ms, Queueing time: mean = 98.528 us, max = 15.090 ms, min = 11.600 us, total = 561.314 ms NodeManagerService.grpc_client.GetResourceLoad - 5697 total (0 active), Execution time: mean = 1.352 ms, total = 7.705 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2289 total (0 active), Execution time: mean = 495.728 us, total = 1.135 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2289 total (0 active), Execution time: mean = 40.221 us, total = 92.065 ms, Queueing time: mean = 90.220 us, max = 601.063 us, min = 10.186 us, total = 206.514 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2279 total (0 active), Execution time: mean = 35.177 us, total = 80.168 ms, Queueing time: mean = 106.860 us, max = 1.496 ms, min = 14.438 us, total = 243.534 ms NodeInfoGcsService.grpc_server.CheckAlive - 2279 total (0 active), Execution time: mean = 517.743 us, total = 1.180 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1901 total (1 active), Execution time: mean = 10.436 us, total = 19.838 ms, Queueing time: mean = 72.823 us, max = 715.636 us, min = 7.259 us, total = 138.436 ms HealthCheck - 1898 total (0 active), Execution time: mean = 7.633 us, total = 14.487 ms, Queueing time: mean = 90.084 us, max = 288.247 us, min = 11.335 us, total = 170.979 ms GcsInMemoryStore.Get - 1197 total (0 active), Execution time: mean = 20.986 us, total = 25.120 ms, Queueing time: mean = 12.676 us, max = 137.899 us, min = 2.227 us, total = 15.173 ms InternalKVGcsService.grpc_server.InternalKVGet - 1196 total (0 active), Execution time: mean = 588.548 us, total = 703.903 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1196 total (0 active), Execution time: mean = 55.809 us, total = 66.747 ms, Queueing time: mean = 149.999 us, max = 22.982 ms, min = 9.184 us, total = 179.399 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1137 total (0 active), Execution time: mean = 629.627 us, total = 715.886 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1137 total (0 active), Execution time: mean = 90.608 us, total = 103.022 ms, Queueing time: mean = 105.893 us, max = 235.695 us, min = 15.511 us, total = 120.401 ms GCSServer.deadline_timer.debug_state_dump - 570 total (1 active), Execution time: mean = 1.371 ms, total = 781.381 ms, Queueing time: mean = 27.059 us, max = 166.240 us, min = -0.023 s, total = 15.424 ms GCSServer.deadline_timer.debug_state_event_stats_print - 95 total (1 active, 1 running), Execution time: mean = 1.542 ms, total = 146.512 ms, Queueing time: mean = 56.185 us, max = 137.733 us, min = 13.943 us, total = 5.338 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:47:15,340 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 263209 total (1 active) Queueing time: mean = 40.999 us, max = 58.960 ms, min = 1.891 us, total = 10.791 s Execution time: mean = 217.060 us, total = 57.132 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 119637 total (0 active), Execution time: mean = 449.719 us, total = 53.803 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 119637 total (0 active), Execution time: mean = 22.877 us, total = 2.737 s, Queueing time: mean = 87.801 us, max = 58.960 ms, min = 2.005 us, total = 10.504 s event_loop_lag_probe - 22793 total (0 active), Execution time: mean = 25.490 us, total = 580.996 ms, Queueing time: mean = 8.868 us, max = 395.058 us, min = 1.891 us, total = 202.133 ms GcsTaskManager.GcJobSummary - 1141 total (1 active), Execution time: mean = 9.696 us, total = 11.063 ms, Queueing time: mean = 74.352 us, max = 3.642 ms, min = 12.167 us, total = 84.835 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:47:15,340 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28080 total (25 active) Queueing time: mean = 15.664 us, max = 600.706 us, min = 1.580 us, total = 439.847 ms Execution time: mean = 4.678 s, total = 131346.854 s Event stats: event_loop_lag_probe - 22793 total (0 active), Execution time: mean = 25.543 us, total = 582.206 ms, Queueing time: mean = 9.212 us, max = 600.706 us, min = 1.580 us, total = 209.962 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1422 total (0 active), Execution time: mean = 36.727 us, total = 52.225 ms, Queueing time: mean = 83.106 us, max = 408.208 us, min = 5.163 us, total = 118.176 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1422 total (24 active), Execution time: mean = 92.367 s, total = 131345.396 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1186 total (0 active), Execution time: mean = 102.588 us, total = 121.670 ms, Queueing time: mean = 91.326 us, max = 236.271 us, min = 11.318 us, total = 108.312 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1186 total (0 active), Execution time: mean = 553.266 us, total = 656.173 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 456.706 us, total = 9.134 ms, Queueing time: mean = 70.240 us, max = 114.456 us, min = 33.142 us, total = 1.405 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:47:15,340 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22840 total (0 active) Queueing time: mean = 10.011 us, max = 2.505 ms, min = 563.000 ns, total = 228.650 ms Execution time: mean = 26.338 us, total = 601.566 ms Event stats: event_loop_lag_probe - 22794 total (0 active), Execution time: mean = 26.230 us, total = 597.882 ms, Queueing time: mean = 9.919 us, max = 2.505 ms, min = 1.873 us, total = 226.094 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:48:15,340 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1149 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1198 - cumulative published bytes: 952533 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:48:15,341 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 69379 total (4 active) Queueing time: mean = 69.651 us, max = 225.583 ms, min = -0.023 s, total = 4.832 s Execution time: mean = 226.700 us, total = 15.728 s Event stats: event_loop_lag_probe - 23032 total (0 active), Execution time: mean = 25.871 us, total = 595.860 ms, Queueing time: mean = 10.002 us, max = 6.895 ms, min = 1.268 us, total = 230.371 ms GcsInMemoryStore.Put - 9257 total (0 active), Execution time: mean = 29.538 us, total = 273.435 ms, Queueing time: mean = 214.479 us, max = 225.278 ms, min = 1.495 us, total = 1.985 s RayletLoadPulled - 5759 total (1 active), Execution time: mean = 244.165 us, total = 1.406 s, Queueing time: mean = 74.440 us, max = 1.351 ms, min = 10.621 us, total = 428.702 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5757 total (0 active), Execution time: mean = 75.194 us, total = 432.894 ms, Queueing time: mean = 98.666 us, max = 15.090 ms, min = 11.600 us, total = 568.019 ms NodeManagerService.grpc_client.GetResourceLoad - 5757 total (0 active), Execution time: mean = 1.350 ms, total = 7.770 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2313 total (0 active), Execution time: mean = 495.377 us, total = 1.146 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2313 total (0 active), Execution time: mean = 40.232 us, total = 93.057 ms, Queueing time: mean = 90.427 us, max = 601.063 us, min = 10.186 us, total = 209.158 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2303 total (0 active), Execution time: mean = 35.191 us, total = 81.046 ms, Queueing time: mean = 106.820 us, max = 1.496 ms, min = 14.438 us, total = 246.007 ms NodeInfoGcsService.grpc_server.CheckAlive - 2303 total (0 active), Execution time: mean = 516.766 us, total = 1.190 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1921 total (1 active), Execution time: mean = 10.423 us, total = 20.023 ms, Queueing time: mean = 72.847 us, max = 715.636 us, min = 7.259 us, total = 139.938 ms HealthCheck - 1918 total (0 active), Execution time: mean = 7.628 us, total = 14.631 ms, Queueing time: mean = 90.176 us, max = 288.247 us, min = 11.335 us, total = 172.958 ms GcsInMemoryStore.Get - 1209 total (0 active), Execution time: mean = 20.971 us, total = 25.354 ms, Queueing time: mean = 12.674 us, max = 137.899 us, min = 2.227 us, total = 15.322 ms InternalKVGcsService.grpc_server.InternalKVGet - 1208 total (0 active), Execution time: mean = 587.242 us, total = 709.388 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1208 total (0 active), Execution time: mean = 55.795 us, total = 67.400 ms, Queueing time: mean = 149.470 us, max = 22.982 ms, min = 9.184 us, total = 180.559 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1149 total (0 active), Execution time: mean = 628.937 us, total = 722.649 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1149 total (0 active), Execution time: mean = 90.566 us, total = 104.060 ms, Queueing time: mean = 106.138 us, max = 235.695 us, min = 15.511 us, total = 121.952 ms GCSServer.deadline_timer.debug_state_dump - 576 total (1 active), Execution time: mean = 1.369 ms, total = 788.410 ms, Queueing time: mean = 27.345 us, max = 166.240 us, min = -0.023 s, total = 15.751 ms GCSServer.deadline_timer.debug_state_event_stats_print - 96 total (1 active, 1 running), Execution time: mean = 1.533 ms, total = 147.167 ms, Queueing time: mean = 56.028 us, max = 137.733 us, min = 13.943 us, total = 5.379 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:48:15,341 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 265981 total (1 active) Queueing time: mean = 40.834 us, max = 58.960 ms, min = 1.891 us, total = 10.861 s Execution time: mean = 216.474 us, total = 57.578 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 120897 total (0 active), Execution time: mean = 448.479 us, total = 54.220 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 120897 total (0 active), Execution time: mean = 22.827 us, total = 2.760 s, Queueing time: mean = 87.438 us, max = 58.960 ms, min = 2.005 us, total = 10.571 s event_loop_lag_probe - 23033 total (0 active), Execution time: mean = 25.494 us, total = 587.200 ms, Queueing time: mean = 8.875 us, max = 395.058 us, min = 1.891 us, total = 204.409 ms GcsTaskManager.GcJobSummary - 1153 total (1 active), Execution time: mean = 9.715 us, total = 11.201 ms, Queueing time: mean = 74.189 us, max = 3.642 ms, min = 12.167 us, total = 85.540 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:48:15,341 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28368 total (25 active) Queueing time: mean = 15.634 us, max = 600.706 us, min = 1.580 us, total = 443.511 ms Execution time: mean = 4.632 s, total = 131407.014 s Event stats: event_loop_lag_probe - 23033 total (0 active), Execution time: mean = 25.544 us, total = 588.354 ms, Queueing time: mean = 9.212 us, max = 600.706 us, min = 1.580 us, total = 212.188 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1434 total (0 active), Execution time: mean = 36.780 us, total = 52.743 ms, Queueing time: mean = 82.899 us, max = 408.208 us, min = 5.163 us, total = 118.877 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1434 total (24 active), Execution time: mean = 91.636 s, total = 131405.543 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1198 total (0 active), Execution time: mean = 102.686 us, total = 123.018 ms, Queueing time: mean = 91.027 us, max = 236.271 us, min = 11.318 us, total = 109.051 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1198 total (0 active), Execution time: mean = 552.452 us, total = 661.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 456.706 us, total = 9.134 ms, Queueing time: mean = 70.240 us, max = 114.456 us, min = 33.142 us, total = 1.405 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:48:15,341 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23080 total (0 active) Queueing time: mean = 10.003 us, max = 2.505 ms, min = 563.000 ns, total = 230.868 ms Execution time: mean = 26.324 us, total = 607.554 ms Event stats: event_loop_lag_probe - 23034 total (0 active), Execution time: mean = 26.216 us, total = 603.869 ms, Queueing time: mean = 9.912 us, max = 2.505 ms, min = 1.873 us, total = 228.312 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:49:15,341 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1161 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1210 - cumulative published bytes: 962361 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:49:15,342 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 70098 total (4 active) Queueing time: mean = 69.292 us, max = 225.583 ms, min = -0.023 s, total = 4.857 s Execution time: mean = 226.573 us, total = 15.882 s Event stats: event_loop_lag_probe - 23272 total (0 active), Execution time: mean = 25.876 us, total = 602.183 ms, Queueing time: mean = 10.000 us, max = 6.895 ms, min = 1.268 us, total = 232.716 ms GcsInMemoryStore.Put - 9353 total (0 active), Execution time: mean = 29.284 us, total = 273.889 ms, Queueing time: mean = 212.735 us, max = 225.278 ms, min = 1.495 us, total = 1.990 s RayletLoadPulled - 5819 total (1 active), Execution time: mean = 244.058 us, total = 1.420 s, Queueing time: mean = 74.370 us, max = 1.351 ms, min = 10.621 us, total = 432.758 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5817 total (0 active), Execution time: mean = 75.112 us, total = 436.928 ms, Queueing time: mean = 98.484 us, max = 15.090 ms, min = 11.600 us, total = 572.883 ms NodeManagerService.grpc_client.GetResourceLoad - 5817 total (0 active), Execution time: mean = 1.349 ms, total = 7.850 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2337 total (0 active), Execution time: mean = 495.036 us, total = 1.157 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2337 total (0 active), Execution time: mean = 40.171 us, total = 93.879 ms, Queueing time: mean = 90.167 us, max = 601.063 us, min = 10.186 us, total = 210.720 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2327 total (0 active), Execution time: mean = 35.153 us, total = 81.801 ms, Queueing time: mean = 106.883 us, max = 1.496 ms, min = 14.438 us, total = 248.718 ms NodeInfoGcsService.grpc_server.CheckAlive - 2327 total (0 active), Execution time: mean = 516.843 us, total = 1.203 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1941 total (1 active), Execution time: mean = 10.418 us, total = 20.222 ms, Queueing time: mean = 72.822 us, max = 715.636 us, min = 7.259 us, total = 141.348 ms HealthCheck - 1938 total (0 active), Execution time: mean = 7.618 us, total = 14.764 ms, Queueing time: mean = 89.998 us, max = 288.247 us, min = 11.335 us, total = 174.416 ms GcsInMemoryStore.Get - 1221 total (0 active), Execution time: mean = 20.924 us, total = 25.548 ms, Queueing time: mean = 12.646 us, max = 137.899 us, min = 2.227 us, total = 15.441 ms InternalKVGcsService.grpc_server.InternalKVGet - 1220 total (0 active), Execution time: mean = 585.946 us, total = 714.855 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1220 total (0 active), Execution time: mean = 55.805 us, total = 68.082 ms, Queueing time: mean = 148.580 us, max = 22.982 ms, min = 9.184 us, total = 181.268 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1161 total (0 active), Execution time: mean = 628.307 us, total = 729.464 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1161 total (0 active), Execution time: mean = 90.495 us, total = 105.065 ms, Queueing time: mean = 105.860 us, max = 235.695 us, min = 15.511 us, total = 122.904 ms GCSServer.deadline_timer.debug_state_dump - 582 total (1 active), Execution time: mean = 1.370 ms, total = 797.477 ms, Queueing time: mean = 27.842 us, max = 166.240 us, min = -0.023 s, total = 16.204 ms GCSServer.deadline_timer.debug_state_event_stats_print - 97 total (1 active, 1 running), Execution time: mean = 1.527 ms, total = 148.149 ms, Queueing time: mean = 55.625 us, max = 137.733 us, min = 13.943 us, total = 5.396 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:49:15,343 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 268753 total (1 active) Queueing time: mean = 40.881 us, max = 58.960 ms, min = 1.891 us, total = 10.987 s Execution time: mean = 216.548 us, total = 58.198 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 122157 total (0 active), Execution time: mean = 448.638 us, total = 54.804 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 122157 total (0 active), Execution time: mean = 22.832 us, total = 2.789 s, Queueing time: mean = 87.543 us, max = 58.960 ms, min = 2.005 us, total = 10.694 s event_loop_lag_probe - 23273 total (0 active), Execution time: mean = 25.493 us, total = 593.294 ms, Queueing time: mean = 8.875 us, max = 395.058 us, min = 1.891 us, total = 206.541 ms GcsTaskManager.GcJobSummary - 1165 total (1 active), Execution time: mean = 9.694 us, total = 11.294 ms, Queueing time: mean = 74.080 us, max = 3.642 ms, min = 12.167 us, total = 86.303 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:49:15,343 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28656 total (25 active) Queueing time: mean = 15.628 us, max = 600.706 us, min = 1.580 us, total = 447.823 ms Execution time: mean = 4.588 s, total = 131467.221 s Event stats: event_loop_lag_probe - 23273 total (0 active), Execution time: mean = 25.531 us, total = 594.186 ms, Queueing time: mean = 9.209 us, max = 600.706 us, min = 1.580 us, total = 214.328 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1446 total (0 active), Execution time: mean = 36.784 us, total = 53.189 ms, Queueing time: mean = 82.965 us, max = 408.208 us, min = 5.163 us, total = 119.968 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1446 total (24 active), Execution time: mean = 90.917 s, total = 131465.735 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1210 total (0 active), Execution time: mean = 102.596 us, total = 124.141 ms, Queueing time: mean = 91.017 us, max = 236.271 us, min = 11.318 us, total = 110.131 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1210 total (0 active), Execution time: mean = 552.463 us, total = 668.480 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 456.706 us, total = 9.134 ms, Queueing time: mean = 70.240 us, max = 114.456 us, min = 33.142 us, total = 1.405 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:49:15,343 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23319 total (0 active) Queueing time: mean = 9.996 us, max = 2.505 ms, min = 563.000 ns, total = 233.108 ms Execution time: mean = 26.314 us, total = 613.608 ms Event stats: event_loop_lag_probe - 23273 total (0 active), Execution time: mean = 26.207 us, total = 609.924 ms, Queueing time: mean = 9.906 us, max = 2.505 ms, min = 1.873 us, total = 230.551 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:50:15,343 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1173 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1222 - cumulative published bytes: 972189 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:50:15,344 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 70817 total (4 active) Queueing time: mean = 68.982 us, max = 225.583 ms, min = -0.023 s, total = 4.885 s Execution time: mean = 226.252 us, total = 16.022 s Event stats: event_loop_lag_probe - 23512 total (0 active), Execution time: mean = 25.863 us, total = 608.100 ms, Queueing time: mean = 9.993 us, max = 6.895 ms, min = 1.268 us, total = 234.944 ms GcsInMemoryStore.Put - 9449 total (0 active), Execution time: mean = 29.039 us, total = 274.389 ms, Queueing time: mean = 211.017 us, max = 225.278 ms, min = 1.495 us, total = 1.994 s RayletLoadPulled - 5879 total (1 active), Execution time: mean = 243.876 us, total = 1.434 s, Queueing time: mean = 74.325 us, max = 1.351 ms, min = 10.621 us, total = 436.955 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5877 total (0 active), Execution time: mean = 75.125 us, total = 441.511 ms, Queueing time: mean = 98.514 us, max = 15.090 ms, min = 11.600 us, total = 578.967 ms NodeManagerService.grpc_client.GetResourceLoad - 5877 total (0 active), Execution time: mean = 1.347 ms, total = 7.919 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2361 total (0 active), Execution time: mean = 494.393 us, total = 1.167 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2361 total (0 active), Execution time: mean = 40.346 us, total = 95.257 ms, Queueing time: mean = 90.103 us, max = 601.063 us, min = 10.186 us, total = 212.732 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2351 total (0 active), Execution time: mean = 35.094 us, total = 82.506 ms, Queueing time: mean = 106.944 us, max = 1.496 ms, min = 14.438 us, total = 251.425 ms NodeInfoGcsService.grpc_server.CheckAlive - 2351 total (0 active), Execution time: mean = 516.417 us, total = 1.214 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1961 total (1 active), Execution time: mean = 10.405 us, total = 20.404 ms, Queueing time: mean = 72.807 us, max = 715.636 us, min = 7.259 us, total = 142.774 ms HealthCheck - 1958 total (0 active), Execution time: mean = 7.610 us, total = 14.900 ms, Queueing time: mean = 90.019 us, max = 288.247 us, min = 11.335 us, total = 176.258 ms GcsInMemoryStore.Get - 1233 total (0 active), Execution time: mean = 20.906 us, total = 25.777 ms, Queueing time: mean = 12.680 us, max = 137.899 us, min = 2.227 us, total = 15.634 ms InternalKVGcsService.grpc_server.InternalKVGet - 1232 total (0 active), Execution time: mean = 584.868 us, total = 720.558 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1232 total (0 active), Execution time: mean = 55.891 us, total = 68.857 ms, Queueing time: mean = 148.034 us, max = 22.982 ms, min = 9.184 us, total = 182.378 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1173 total (0 active), Execution time: mean = 627.819 us, total = 736.432 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1173 total (0 active), Execution time: mean = 90.448 us, total = 106.096 ms, Queueing time: mean = 106.026 us, max = 235.695 us, min = 15.511 us, total = 124.368 ms GCSServer.deadline_timer.debug_state_dump - 588 total (1 active), Execution time: mean = 1.366 ms, total = 803.108 ms, Queueing time: mean = 28.114 us, max = 166.240 us, min = -0.023 s, total = 16.531 ms GCSServer.deadline_timer.debug_state_event_stats_print - 98 total (1 active, 1 running), Execution time: mean = 1.530 ms, total = 149.939 ms, Queueing time: mean = 55.952 us, max = 137.733 us, min = 13.943 us, total = 5.483 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:50:15,344 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 271525 total (1 active) Queueing time: mean = 40.771 us, max = 58.960 ms, min = 1.891 us, total = 11.070 s Execution time: mean = 216.128 us, total = 58.684 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 123417 total (0 active), Execution time: mean = 447.747 us, total = 55.260 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 123417 total (0 active), Execution time: mean = 22.800 us, total = 2.814 s, Queueing time: mean = 87.303 us, max = 58.960 ms, min = 2.005 us, total = 10.775 s event_loop_lag_probe - 23513 total (0 active), Execution time: mean = 25.485 us, total = 599.228 ms, Queueing time: mean = 8.873 us, max = 395.058 us, min = 1.891 us, total = 208.636 ms GcsTaskManager.GcJobSummary - 1177 total (1 active), Execution time: mean = 9.687 us, total = 11.401 ms, Queueing time: mean = 73.904 us, max = 3.642 ms, min = 12.167 us, total = 86.984 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:50:15,344 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28944 total (25 active) Queueing time: mean = 15.596 us, max = 600.706 us, min = 1.580 us, total = 451.418 ms Execution time: mean = 4.544 s, total = 131527.293 s Event stats: event_loop_lag_probe - 23513 total (0 active), Execution time: mean = 25.518 us, total = 600.002 ms, Queueing time: mean = 9.206 us, max = 600.706 us, min = 1.580 us, total = 216.466 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1458 total (0 active), Execution time: mean = 36.740 us, total = 53.568 ms, Queueing time: mean = 82.757 us, max = 408.208 us, min = 5.163 us, total = 120.659 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1458 total (24 active), Execution time: mean = 90.210 s, total = 131525.795 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1222 total (0 active), Execution time: mean = 102.457 us, total = 125.203 ms, Queueing time: mean = 90.750 us, max = 236.271 us, min = 11.318 us, total = 110.897 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1222 total (0 active), Execution time: mean = 551.460 us, total = 673.884 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 456.706 us, total = 9.134 ms, Queueing time: mean = 70.240 us, max = 114.456 us, min = 33.142 us, total = 1.405 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:50:15,344 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23559 total (0 active) Queueing time: mean = 9.989 us, max = 2.505 ms, min = 563.000 ns, total = 235.336 ms Execution time: mean = 26.299 us, total = 619.579 ms Event stats: event_loop_lag_probe - 23513 total (0 active), Execution time: mean = 26.194 us, total = 615.895 ms, Queueing time: mean = 9.900 us, max = 2.505 ms, min = 1.873 us, total = 232.779 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:51:15,344 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1185 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1235 - cumulative published bytes: 982051 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:51:15,345 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 71536 total (4 active) Queueing time: mean = 68.721 us, max = 225.583 ms, min = -0.023 s, total = 4.916 s Execution time: mean = 226.435 us, total = 16.198 s Event stats: event_loop_lag_probe - 23752 total (0 active), Execution time: mean = 25.891 us, total = 614.974 ms, Queueing time: mean = 10.001 us, max = 6.895 ms, min = 1.268 us, total = 237.535 ms GcsInMemoryStore.Put - 9545 total (0 active), Execution time: mean = 28.799 us, total = 274.885 ms, Queueing time: mean = 209.336 us, max = 225.278 ms, min = 1.495 us, total = 1.998 s RayletLoadPulled - 5939 total (1 active), Execution time: mean = 243.958 us, total = 1.449 s, Queueing time: mean = 74.474 us, max = 1.351 ms, min = 10.621 us, total = 442.301 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5937 total (0 active), Execution time: mean = 75.163 us, total = 446.243 ms, Queueing time: mean = 98.649 us, max = 15.090 ms, min = 11.600 us, total = 585.676 ms NodeManagerService.grpc_client.GetResourceLoad - 5937 total (0 active), Execution time: mean = 1.349 ms, total = 8.011 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2385 total (0 active), Execution time: mean = 495.089 us, total = 1.181 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2385 total (0 active), Execution time: mean = 40.353 us, total = 96.242 ms, Queueing time: mean = 90.225 us, max = 601.063 us, min = 10.186 us, total = 215.186 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2375 total (0 active), Execution time: mean = 35.137 us, total = 83.450 ms, Queueing time: mean = 107.056 us, max = 1.496 ms, min = 14.438 us, total = 254.257 ms NodeInfoGcsService.grpc_server.CheckAlive - 2375 total (0 active), Execution time: mean = 517.041 us, total = 1.228 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1981 total (1 active), Execution time: mean = 10.406 us, total = 20.614 ms, Queueing time: mean = 72.951 us, max = 715.636 us, min = 7.259 us, total = 144.516 ms HealthCheck - 1978 total (0 active), Execution time: mean = 7.607 us, total = 15.046 ms, Queueing time: mean = 90.051 us, max = 288.247 us, min = 11.335 us, total = 178.122 ms GcsInMemoryStore.Get - 1245 total (0 active), Execution time: mean = 20.933 us, total = 26.062 ms, Queueing time: mean = 12.686 us, max = 137.899 us, min = 2.227 us, total = 15.794 ms InternalKVGcsService.grpc_server.InternalKVGet - 1244 total (0 active), Execution time: mean = 584.977 us, total = 727.711 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1244 total (0 active), Execution time: mean = 55.920 us, total = 69.565 ms, Queueing time: mean = 147.556 us, max = 22.982 ms, min = 9.184 us, total = 183.560 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1185 total (0 active), Execution time: mean = 628.506 us, total = 744.779 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1185 total (0 active), Execution time: mean = 90.430 us, total = 107.160 ms, Queueing time: mean = 106.108 us, max = 235.695 us, min = 15.511 us, total = 125.738 ms GCSServer.deadline_timer.debug_state_dump - 594 total (1 active), Execution time: mean = 1.366 ms, total = 811.297 ms, Queueing time: mean = 28.576 us, max = 166.240 us, min = -0.023 s, total = 16.974 ms GCSServer.deadline_timer.debug_state_event_stats_print - 99 total (1 active, 1 running), Execution time: mean = 1.524 ms, total = 150.892 ms, Queueing time: mean = 55.575 us, max = 137.733 us, min = 13.943 us, total = 5.502 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:51:15,345 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 274297 total (1 active) Queueing time: mean = 40.882 us, max = 58.960 ms, min = 1.891 us, total = 11.214 s Execution time: mean = 216.664 us, total = 59.430 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 124677 total (0 active), Execution time: mean = 448.868 us, total = 55.963 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 124677 total (0 active), Execution time: mean = 22.852 us, total = 2.849 s, Queueing time: mean = 87.547 us, max = 58.960 ms, min = 2.005 us, total = 10.915 s event_loop_lag_probe - 23753 total (0 active), Execution time: mean = 25.516 us, total = 606.089 ms, Queueing time: mean = 8.880 us, max = 395.058 us, min = 1.891 us, total = 210.920 ms GcsTaskManager.GcJobSummary - 1189 total (1 active), Execution time: mean = 9.673 us, total = 11.501 ms, Queueing time: mean = 73.800 us, max = 3.642 ms, min = 12.167 us, total = 87.748 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:51:15,345 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29236 total (25 active) Queueing time: mean = 15.615 us, max = 600.706 us, min = 1.580 us, total = 456.508 ms Execution time: mean = 4.501 s, total = 131587.540 s Event stats: event_loop_lag_probe - 23753 total (0 active), Execution time: mean = 25.543 us, total = 606.721 ms, Queueing time: mean = 9.215 us, max = 600.706 us, min = 1.580 us, total = 218.872 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1471 total (0 active), Execution time: mean = 36.819 us, total = 54.161 ms, Queueing time: mean = 82.965 us, max = 408.208 us, min = 5.163 us, total = 122.042 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1471 total (24 active), Execution time: mean = 89.453 s, total = 131586.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1235 total (0 active), Execution time: mean = 102.442 us, total = 126.516 ms, Queueing time: mean = 90.849 us, max = 236.271 us, min = 11.318 us, total = 112.198 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1235 total (0 active), Execution time: mean = 552.324 us, total = 682.121 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 456.706 us, total = 9.134 ms, Queueing time: mean = 70.240 us, max = 114.456 us, min = 33.142 us, total = 1.405 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:51:15,346 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23799 total (0 active) Queueing time: mean = 9.993 us, max = 2.505 ms, min = 563.000 ns, total = 237.822 ms Execution time: mean = 26.314 us, total = 626.253 ms Event stats: event_loop_lag_probe - 23753 total (0 active), Execution time: mean = 26.210 us, total = 622.569 ms, Queueing time: mean = 9.905 us, max = 2.505 ms, min = 1.873 us, total = 235.266 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:52:15,346 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1197 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1247 - cumulative published bytes: 991879 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:52:15,347 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 72255 total (4 active) Queueing time: mean = 68.431 us, max = 225.583 ms, min = -0.023 s, total = 4.944 s Execution time: mean = 226.532 us, total = 16.368 s Event stats: event_loop_lag_probe - 23992 total (0 active), Execution time: mean = 25.911 us, total = 621.655 ms, Queueing time: mean = 10.005 us, max = 6.895 ms, min = 1.268 us, total = 240.047 ms GcsInMemoryStore.Put - 9641 total (0 active), Execution time: mean = 28.564 us, total = 275.385 ms, Queueing time: mean = 207.681 us, max = 225.278 ms, min = 1.495 us, total = 2.002 s RayletLoadPulled - 5999 total (1 active), Execution time: mean = 244.005 us, total = 1.464 s, Queueing time: mean = 74.457 us, max = 1.351 ms, min = 10.621 us, total = 446.665 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5997 total (0 active), Execution time: mean = 75.146 us, total = 450.650 ms, Queueing time: mean = 98.602 us, max = 15.090 ms, min = 11.600 us, total = 591.315 ms NodeManagerService.grpc_client.GetResourceLoad - 5997 total (0 active), Execution time: mean = 1.351 ms, total = 8.101 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2409 total (0 active), Execution time: mean = 495.254 us, total = 1.193 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2409 total (0 active), Execution time: mean = 40.314 us, total = 97.117 ms, Queueing time: mean = 90.300 us, max = 601.063 us, min = 10.186 us, total = 217.532 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2399 total (0 active), Execution time: mean = 35.085 us, total = 84.169 ms, Queueing time: mean = 106.958 us, max = 1.496 ms, min = 14.438 us, total = 256.593 ms NodeInfoGcsService.grpc_server.CheckAlive - 2399 total (0 active), Execution time: mean = 516.985 us, total = 1.240 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2001 total (1 active), Execution time: mean = 10.467 us, total = 20.945 ms, Queueing time: mean = 73.159 us, max = 715.636 us, min = 7.259 us, total = 146.392 ms HealthCheck - 1998 total (0 active), Execution time: mean = 7.612 us, total = 15.208 ms, Queueing time: mean = 90.220 us, max = 288.247 us, min = 11.335 us, total = 180.260 ms GcsInMemoryStore.Get - 1257 total (0 active), Execution time: mean = 20.911 us, total = 26.285 ms, Queueing time: mean = 12.677 us, max = 137.899 us, min = 2.227 us, total = 15.935 ms InternalKVGcsService.grpc_server.InternalKVGet - 1256 total (0 active), Execution time: mean = 584.866 us, total = 734.592 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1256 total (0 active), Execution time: mean = 55.872 us, total = 70.175 ms, Queueing time: mean = 147.088 us, max = 22.982 ms, min = 9.184 us, total = 184.742 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1197 total (0 active), Execution time: mean = 628.676 us, total = 752.525 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1197 total (0 active), Execution time: mean = 90.338 us, total = 108.134 ms, Queueing time: mean = 106.167 us, max = 235.695 us, min = 15.511 us, total = 127.081 ms GCSServer.deadline_timer.debug_state_dump - 600 total (1 active), Execution time: mean = 1.366 ms, total = 819.813 ms, Queueing time: mean = 28.899 us, max = 166.240 us, min = -0.023 s, total = 17.340 ms GCSServer.deadline_timer.debug_state_event_stats_print - 100 total (1 active, 1 running), Execution time: mean = 1.524 ms, total = 152.396 ms, Queueing time: mean = 55.482 us, max = 137.733 us, min = 13.943 us, total = 5.548 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:52:15,347 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 277069 total (1 active) Queueing time: mean = 40.935 us, max = 58.960 ms, min = 1.891 us, total = 11.342 s Execution time: mean = 216.850 us, total = 60.082 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 125937 total (0 active), Execution time: mean = 449.252 us, total = 56.577 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 125937 total (0 active), Execution time: mean = 22.872 us, total = 2.880 s, Queueing time: mean = 87.663 us, max = 58.960 ms, min = 2.005 us, total = 11.040 s event_loop_lag_probe - 23993 total (0 active), Execution time: mean = 25.539 us, total = 612.757 ms, Queueing time: mean = 8.884 us, max = 395.058 us, min = 1.891 us, total = 213.155 ms GcsTaskManager.GcJobSummary - 1201 total (1 active), Execution time: mean = 9.669 us, total = 11.613 ms, Queueing time: mean = 73.842 us, max = 3.642 ms, min = 12.167 us, total = 88.684 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:52:15,347 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29561 total (25 active) Queueing time: mean = 15.615 us, max = 600.706 us, min = 1.580 us, total = 461.584 ms Execution time: mean = 4.819 s, total = 142447.662 s Event stats: event_loop_lag_probe - 23993 total (0 active), Execution time: mean = 25.558 us, total = 613.214 ms, Queueing time: mean = 9.220 us, max = 600.706 us, min = 1.580 us, total = 221.223 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1501 total (0 active), Execution time: mean = 36.565 us, total = 54.885 ms, Queueing time: mean = 82.340 us, max = 408.208 us, min = 4.084 us, total = 123.592 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1501 total (24 active), Execution time: mean = 94.901 s, total = 142446.132 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1247 total (0 active), Execution time: mean = 102.335 us, total = 127.612 ms, Queueing time: mean = 90.882 us, max = 236.271 us, min = 11.318 us, total = 113.330 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1247 total (0 active), Execution time: mean = 552.504 us, total = 688.972 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 456.864 us, total = 9.594 ms, Queueing time: mean = 68.967 us, max = 114.456 us, min = 33.142 us, total = 1.448 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:52:15,347 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24039 total (0 active) Queueing time: mean = 9.999 us, max = 2.505 ms, min = 563.000 ns, total = 240.366 ms Execution time: mean = 26.324 us, total = 632.801 ms Event stats: event_loop_lag_probe - 23993 total (0 active), Execution time: mean = 26.221 us, total = 629.117 ms, Queueing time: mean = 9.912 us, max = 2.505 ms, min = 1.873 us, total = 237.810 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:53:15,348 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1209 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1259 - cumulative published bytes: 1001707 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:53:15,348 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 72974 total (4 active) Queueing time: mean = 68.132 us, max = 225.583 ms, min = -0.023 s, total = 4.972 s Execution time: mean = 226.365 us, total = 16.519 s Event stats: event_loop_lag_probe - 24232 total (0 active), Execution time: mean = 25.909 us, total = 627.815 ms, Queueing time: mean = 10.001 us, max = 6.895 ms, min = 1.268 us, total = 242.347 ms GcsInMemoryStore.Put - 9737 total (0 active), Execution time: mean = 28.331 us, total = 275.862 ms, Queueing time: mean = 206.044 us, max = 225.278 ms, min = 1.495 us, total = 2.006 s RayletLoadPulled - 6059 total (1 active), Execution time: mean = 243.896 us, total = 1.478 s, Queueing time: mean = 74.417 us, max = 1.351 ms, min = 10.621 us, total = 450.895 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6057 total (0 active), Execution time: mean = 75.250 us, total = 455.792 ms, Queueing time: mean = 98.566 us, max = 15.090 ms, min = 11.600 us, total = 597.013 ms NodeManagerService.grpc_client.GetResourceLoad - 6057 total (0 active), Execution time: mean = 1.350 ms, total = 8.178 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2433 total (0 active), Execution time: mean = 494.776 us, total = 1.204 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2433 total (0 active), Execution time: mean = 40.382 us, total = 98.251 ms, Queueing time: mean = 90.320 us, max = 601.063 us, min = 10.186 us, total = 219.749 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2423 total (0 active), Execution time: mean = 35.101 us, total = 85.050 ms, Queueing time: mean = 106.835 us, max = 1.496 ms, min = 14.438 us, total = 258.862 ms NodeInfoGcsService.grpc_server.CheckAlive - 2423 total (0 active), Execution time: mean = 516.313 us, total = 1.251 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2021 total (1 active), Execution time: mean = 10.457 us, total = 21.135 ms, Queueing time: mean = 73.143 us, max = 715.636 us, min = 7.259 us, total = 147.822 ms HealthCheck - 2018 total (0 active), Execution time: mean = 7.610 us, total = 15.358 ms, Queueing time: mean = 90.401 us, max = 288.247 us, min = 11.335 us, total = 182.430 ms GcsInMemoryStore.Get - 1269 total (0 active), Execution time: mean = 20.892 us, total = 26.512 ms, Queueing time: mean = 12.673 us, max = 137.899 us, min = 2.227 us, total = 16.083 ms InternalKVGcsService.grpc_server.InternalKVGet - 1268 total (0 active), Execution time: mean = 583.878 us, total = 740.357 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1268 total (0 active), Execution time: mean = 55.874 us, total = 70.848 ms, Queueing time: mean = 146.662 us, max = 22.982 ms, min = 9.184 us, total = 185.967 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1209 total (0 active), Execution time: mean = 627.609 us, total = 758.779 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1209 total (0 active), Execution time: mean = 90.381 us, total = 109.271 ms, Queueing time: mean = 106.072 us, max = 235.695 us, min = 15.511 us, total = 128.241 ms GCSServer.deadline_timer.debug_state_dump - 606 total (1 active), Execution time: mean = 1.367 ms, total = 828.206 ms, Queueing time: mean = 29.310 us, max = 166.240 us, min = -0.023 s, total = 17.762 ms GCSServer.deadline_timer.debug_state_event_stats_print - 101 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 154.033 ms, Queueing time: mean = 56.216 us, max = 137.733 us, min = 13.943 us, total = 5.678 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:53:15,348 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 279840 total (1 active) Queueing time: mean = 40.826 us, max = 58.960 ms, min = 1.891 us, total = 11.425 s Execution time: mean = 216.493 us, total = 60.583 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 127197 total (0 active), Execution time: mean = 448.499 us, total = 57.048 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 127197 total (0 active), Execution time: mean = 22.841 us, total = 2.905 s, Queueing time: mean = 87.420 us, max = 58.960 ms, min = 2.005 us, total = 11.120 s event_loop_lag_probe - 24232 total (0 active), Execution time: mean = 25.530 us, total = 618.638 ms, Queueing time: mean = 8.880 us, max = 395.058 us, min = 1.891 us, total = 215.175 ms GcsTaskManager.GcJobSummary - 1213 total (1 active), Execution time: mean = 9.661 us, total = 11.719 ms, Queueing time: mean = 74.185 us, max = 3.642 ms, min = 12.167 us, total = 89.986 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:53:15,348 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29849 total (25 active) Queueing time: mean = 15.598 us, max = 600.706 us, min = 1.580 us, total = 465.570 ms Execution time: mean = 4.774 s, total = 142507.819 s Event stats: event_loop_lag_probe - 24233 total (0 active), Execution time: mean = 25.554 us, total = 619.256 ms, Queueing time: mean = 9.217 us, max = 600.706 us, min = 1.580 us, total = 223.346 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1513 total (0 active), Execution time: mean = 36.548 us, total = 55.297 ms, Queueing time: mean = 82.196 us, max = 408.208 us, min = 4.084 us, total = 124.362 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1513 total (24 active), Execution time: mean = 94.188 s, total = 142506.276 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1259 total (0 active), Execution time: mean = 102.197 us, total = 128.666 ms, Queueing time: mean = 90.883 us, max = 236.271 us, min = 11.318 us, total = 114.422 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1259 total (0 active), Execution time: mean = 552.028 us, total = 695.004 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 456.864 us, total = 9.594 ms, Queueing time: mean = 68.967 us, max = 114.456 us, min = 33.142 us, total = 1.448 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:53:15,348 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24279 total (0 active) Queueing time: mean = 9.993 us, max = 2.505 ms, min = 563.000 ns, total = 242.629 ms Execution time: mean = 26.314 us, total = 638.880 ms Event stats: event_loop_lag_probe - 24233 total (0 active), Execution time: mean = 26.212 us, total = 635.196 ms, Queueing time: mean = 9.907 us, max = 2.505 ms, min = 1.873 us, total = 240.072 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:54:15,349 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1221 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1271 - cumulative published bytes: 1011535 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:54:15,349 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 73693 total (4 active) Queueing time: mean = 67.819 us, max = 225.583 ms, min = -0.023 s, total = 4.998 s Execution time: mean = 226.356 us, total = 16.681 s Event stats: event_loop_lag_probe - 24472 total (0 active), Execution time: mean = 25.907 us, total = 633.985 ms, Queueing time: mean = 10.000 us, max = 6.895 ms, min = 1.268 us, total = 244.727 ms GcsInMemoryStore.Put - 9833 total (0 active), Execution time: mean = 28.109 us, total = 276.398 ms, Queueing time: mean = 204.428 us, max = 225.278 ms, min = 1.495 us, total = 2.010 s RayletLoadPulled - 6119 total (1 active), Execution time: mean = 243.823 us, total = 1.492 s, Queueing time: mean = 74.360 us, max = 1.351 ms, min = 10.621 us, total = 455.006 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6117 total (0 active), Execution time: mean = 75.266 us, total = 460.400 ms, Queueing time: mean = 98.480 us, max = 15.090 ms, min = 11.600 us, total = 602.403 ms NodeManagerService.grpc_client.GetResourceLoad - 6117 total (0 active), Execution time: mean = 1.351 ms, total = 8.265 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2457 total (0 active), Execution time: mean = 494.724 us, total = 1.216 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2457 total (0 active), Execution time: mean = 40.353 us, total = 99.146 ms, Queueing time: mean = 90.316 us, max = 601.063 us, min = 10.186 us, total = 221.907 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2447 total (0 active), Execution time: mean = 35.132 us, total = 85.968 ms, Queueing time: mean = 106.550 us, max = 1.496 ms, min = 14.438 us, total = 260.729 ms NodeInfoGcsService.grpc_server.CheckAlive - 2447 total (0 active), Execution time: mean = 515.967 us, total = 1.263 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2041 total (1 active), Execution time: mean = 10.448 us, total = 21.325 ms, Queueing time: mean = 73.124 us, max = 715.636 us, min = 7.259 us, total = 149.246 ms HealthCheck - 2038 total (0 active), Execution time: mean = 7.602 us, total = 15.493 ms, Queueing time: mean = 90.301 us, max = 288.247 us, min = 11.335 us, total = 184.034 ms GcsInMemoryStore.Get - 1281 total (0 active), Execution time: mean = 21.005 us, total = 26.908 ms, Queueing time: mean = 12.655 us, max = 137.899 us, min = 2.227 us, total = 16.212 ms InternalKVGcsService.grpc_server.InternalKVGet - 1280 total (0 active), Execution time: mean = 583.355 us, total = 746.694 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1280 total (0 active), Execution time: mean = 55.812 us, total = 71.439 ms, Queueing time: mean = 146.252 us, max = 22.982 ms, min = 9.184 us, total = 187.202 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1221 total (0 active), Execution time: mean = 626.524 us, total = 764.986 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1221 total (0 active), Execution time: mean = 90.285 us, total = 110.237 ms, Queueing time: mean = 106.102 us, max = 235.695 us, min = 15.511 us, total = 129.550 ms GCSServer.deadline_timer.debug_state_dump - 612 total (1 active), Execution time: mean = 1.368 ms, total = 837.358 ms, Queueing time: mean = 29.594 us, max = 166.240 us, min = -0.023 s, total = 18.112 ms GCSServer.deadline_timer.debug_state_event_stats_print - 102 total (1 active, 1 running), Execution time: mean = 1.520 ms, total = 155.053 ms, Queueing time: mean = 56.023 us, max = 137.733 us, min = 13.943 us, total = 5.714 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:54:15,349 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 282612 total (1 active) Queueing time: mean = 40.826 us, max = 58.960 ms, min = 1.891 us, total = 11.538 s Execution time: mean = 216.530 us, total = 61.194 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 128457 total (0 active), Execution time: mean = 448.574 us, total = 57.623 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 128457 total (0 active), Execution time: mean = 22.846 us, total = 2.935 s, Queueing time: mean = 87.418 us, max = 58.960 ms, min = 2.005 us, total = 11.229 s event_loop_lag_probe - 24472 total (0 active), Execution time: mean = 25.536 us, total = 624.921 ms, Queueing time: mean = 8.880 us, max = 395.058 us, min = 1.891 us, total = 217.308 ms GcsTaskManager.GcJobSummary - 1225 total (1 active), Execution time: mean = 9.651 us, total = 11.822 ms, Queueing time: mean = 74.351 us, max = 3.642 ms, min = 12.167 us, total = 91.080 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:54:15,349 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30137 total (25 active) Queueing time: mean = 15.603 us, max = 600.706 us, min = 1.580 us, total = 470.241 ms Execution time: mean = 4.731 s, total = 142568.022 s Event stats: event_loop_lag_probe - 24473 total (0 active), Execution time: mean = 25.546 us, total = 625.190 ms, Queueing time: mean = 9.216 us, max = 600.706 us, min = 1.580 us, total = 225.533 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1525 total (0 active), Execution time: mean = 36.565 us, total = 55.761 ms, Queueing time: mean = 82.363 us, max = 408.208 us, min = 4.084 us, total = 125.604 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1525 total (24 active), Execution time: mean = 93.486 s, total = 142566.464 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1271 total (0 active), Execution time: mean = 102.129 us, total = 129.806 ms, Queueing time: mean = 91.003 us, max = 236.271 us, min = 11.318 us, total = 115.665 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1271 total (0 active), Execution time: mean = 551.959 us, total = 701.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 456.864 us, total = 9.594 ms, Queueing time: mean = 68.967 us, max = 114.456 us, min = 33.142 us, total = 1.448 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:54:15,349 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24519 total (0 active) Queueing time: mean = 9.988 us, max = 2.505 ms, min = 563.000 ns, total = 244.885 ms Execution time: mean = 26.302 us, total = 644.897 ms Event stats: event_loop_lag_probe - 24473 total (0 active), Execution time: mean = 26.201 us, total = 641.213 ms, Queueing time: mean = 9.902 us, max = 2.505 ms, min = 1.873 us, total = 242.329 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:55:15,349 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1233 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1283 - cumulative published bytes: 1021363 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:55:15,350 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 74412 total (4 active) Queueing time: mean = 67.555 us, max = 225.583 ms, min = -0.023 s, total = 5.027 s Execution time: mean = 226.114 us, total = 16.826 s Event stats: event_loop_lag_probe - 24712 total (0 active), Execution time: mean = 25.914 us, total = 640.384 ms, Queueing time: mean = 9.999 us, max = 6.895 ms, min = 1.268 us, total = 247.091 ms GcsInMemoryStore.Put - 9929 total (0 active), Execution time: mean = 27.888 us, total = 276.896 ms, Queueing time: mean = 202.880 us, max = 225.278 ms, min = 1.495 us, total = 2.014 s RayletLoadPulled - 6179 total (1 active), Execution time: mean = 243.618 us, total = 1.505 s, Queueing time: mean = 74.328 us, max = 1.351 ms, min = 10.621 us, total = 459.274 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6177 total (0 active), Execution time: mean = 75.235 us, total = 464.728 ms, Queueing time: mean = 98.578 us, max = 15.090 ms, min = 11.600 us, total = 608.919 ms NodeManagerService.grpc_client.GetResourceLoad - 6177 total (0 active), Execution time: mean = 1.350 ms, total = 8.337 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2481 total (0 active), Execution time: mean = 494.589 us, total = 1.227 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2481 total (0 active), Execution time: mean = 40.354 us, total = 100.117 ms, Queueing time: mean = 90.453 us, max = 601.063 us, min = 10.186 us, total = 224.414 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2471 total (0 active), Execution time: mean = 35.086 us, total = 86.698 ms, Queueing time: mean = 106.526 us, max = 1.496 ms, min = 14.438 us, total = 263.225 ms NodeInfoGcsService.grpc_server.CheckAlive - 2471 total (0 active), Execution time: mean = 515.281 us, total = 1.273 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2061 total (1 active), Execution time: mean = 10.432 us, total = 21.499 ms, Queueing time: mean = 73.043 us, max = 715.636 us, min = 7.259 us, total = 150.542 ms HealthCheck - 2058 total (0 active), Execution time: mean = 7.600 us, total = 15.640 ms, Queueing time: mean = 90.472 us, max = 288.247 us, min = 11.335 us, total = 186.191 ms GcsInMemoryStore.Get - 1293 total (0 active), Execution time: mean = 20.980 us, total = 27.127 ms, Queueing time: mean = 12.653 us, max = 137.899 us, min = 2.227 us, total = 16.360 ms InternalKVGcsService.grpc_server.InternalKVGet - 1292 total (0 active), Execution time: mean = 582.880 us, total = 753.082 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1292 total (0 active), Execution time: mean = 55.783 us, total = 72.071 ms, Queueing time: mean = 145.827 us, max = 22.982 ms, min = 9.184 us, total = 188.409 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1233 total (0 active), Execution time: mean = 626.586 us, total = 772.580 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1233 total (0 active), Execution time: mean = 90.266 us, total = 111.298 ms, Queueing time: mean = 106.233 us, max = 235.695 us, min = 15.511 us, total = 130.985 ms GCSServer.deadline_timer.debug_state_dump - 618 total (1 active), Execution time: mean = 1.367 ms, total = 844.687 ms, Queueing time: mean = 30.012 us, max = 166.240 us, min = -0.023 s, total = 18.547 ms GCSServer.deadline_timer.debug_state_event_stats_print - 103 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 155.955 ms, Queueing time: mean = 55.829 us, max = 137.733 us, min = 13.943 us, total = 5.750 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:55:15,351 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 285384 total (1 active) Queueing time: mean = 40.737 us, max = 58.960 ms, min = 1.891 us, total = 11.626 s Execution time: mean = 216.220 us, total = 61.706 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 129717 total (0 active), Execution time: mean = 447.908 us, total = 58.101 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 129717 total (0 active), Execution time: mean = 22.828 us, total = 2.961 s, Queueing time: mean = 87.222 us, max = 58.960 ms, min = 2.005 us, total = 11.314 s event_loop_lag_probe - 24712 total (0 active), Execution time: mean = 25.550 us, total = 631.400 ms, Queueing time: mean = 8.882 us, max = 395.058 us, min = 1.891 us, total = 219.499 ms GcsTaskManager.GcJobSummary - 1237 total (1 active), Execution time: mean = 9.638 us, total = 11.923 ms, Queueing time: mean = 74.317 us, max = 3.642 ms, min = 12.167 us, total = 91.931 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:55:15,351 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30425 total (25 active) Queueing time: mean = 15.579 us, max = 600.706 us, min = 1.580 us, total = 473.990 ms Execution time: mean = 4.688 s, total = 142628.165 s Event stats: event_loop_lag_probe - 24713 total (0 active), Execution time: mean = 25.525 us, total = 630.799 ms, Queueing time: mean = 9.210 us, max = 600.706 us, min = 1.580 us, total = 227.615 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1537 total (0 active), Execution time: mean = 36.609 us, total = 56.268 ms, Queueing time: mean = 82.219 us, max = 408.208 us, min = 4.084 us, total = 126.370 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1537 total (24 active), Execution time: mean = 92.795 s, total = 142626.593 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1283 total (0 active), Execution time: mean = 102.207 us, total = 131.131 ms, Queueing time: mean = 90.853 us, max = 236.271 us, min = 11.318 us, total = 116.565 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1283 total (0 active), Execution time: mean = 551.520 us, total = 707.600 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 456.864 us, total = 9.594 ms, Queueing time: mean = 68.967 us, max = 114.456 us, min = 33.142 us, total = 1.448 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:55:15,351 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24759 total (0 active) Queueing time: mean = 9.988 us, max = 2.505 ms, min = 563.000 ns, total = 247.291 ms Execution time: mean = 26.297 us, total = 651.088 ms Event stats: event_loop_lag_probe - 24713 total (0 active), Execution time: mean = 26.197 us, total = 647.404 ms, Queueing time: mean = 9.903 us, max = 2.505 ms, min = 1.873 us, total = 244.735 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:56:15,351 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1245 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1295 - cumulative published bytes: 1031191 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:56:15,352 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 75130 total (4 active) Queueing time: mean = 67.246 us, max = 225.583 ms, min = -0.023 s, total = 5.052 s Execution time: mean = 225.663 us, total = 16.954 s Event stats: event_loop_lag_probe - 24951 total (0 active), Execution time: mean = 25.892 us, total = 646.021 ms, Queueing time: mean = 9.989 us, max = 6.895 ms, min = 1.268 us, total = 249.224 ms GcsInMemoryStore.Put - 10025 total (0 active), Execution time: mean = 27.660 us, total = 277.289 ms, Queueing time: mean = 201.311 us, max = 225.278 ms, min = 1.495 us, total = 2.018 s RayletLoadPulled - 6239 total (1 active), Execution time: mean = 243.516 us, total = 1.519 s, Queueing time: mean = 74.283 us, max = 1.351 ms, min = 10.621 us, total = 463.452 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6237 total (0 active), Execution time: mean = 75.184 us, total = 468.922 ms, Queueing time: mean = 98.553 us, max = 15.090 ms, min = 11.600 us, total = 614.674 ms NodeManagerService.grpc_client.GetResourceLoad - 6237 total (0 active), Execution time: mean = 1.347 ms, total = 8.401 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2505 total (0 active), Execution time: mean = 492.950 us, total = 1.235 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2505 total (0 active), Execution time: mean = 40.231 us, total = 100.778 ms, Queueing time: mean = 90.336 us, max = 601.063 us, min = 10.186 us, total = 226.291 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2495 total (0 active), Execution time: mean = 35.028 us, total = 87.395 ms, Queueing time: mean = 106.425 us, max = 1.496 ms, min = 14.438 us, total = 265.530 ms NodeInfoGcsService.grpc_server.CheckAlive - 2495 total (0 active), Execution time: mean = 513.937 us, total = 1.282 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2081 total (1 active), Execution time: mean = 10.406 us, total = 21.656 ms, Queueing time: mean = 72.829 us, max = 715.636 us, min = 7.259 us, total = 151.557 ms HealthCheck - 2078 total (0 active), Execution time: mean = 7.584 us, total = 15.759 ms, Queueing time: mean = 90.389 us, max = 288.247 us, min = 11.335 us, total = 187.829 ms GcsInMemoryStore.Get - 1305 total (0 active), Execution time: mean = 20.935 us, total = 27.320 ms, Queueing time: mean = 12.631 us, max = 137.899 us, min = 2.227 us, total = 16.483 ms InternalKVGcsService.grpc_server.InternalKVGet - 1304 total (0 active), Execution time: mean = 580.843 us, total = 757.419 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1304 total (0 active), Execution time: mean = 55.686 us, total = 72.615 ms, Queueing time: mean = 145.128 us, max = 22.982 ms, min = 9.184 us, total = 189.247 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1245 total (0 active), Execution time: mean = 624.992 us, total = 778.115 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1245 total (0 active), Execution time: mean = 90.281 us, total = 112.400 ms, Queueing time: mean = 106.117 us, max = 235.695 us, min = 15.511 us, total = 132.116 ms GCSServer.deadline_timer.debug_state_dump - 624 total (1 active), Execution time: mean = 1.366 ms, total = 852.660 ms, Queueing time: mean = 30.595 us, max = 178.374 us, min = -0.023 s, total = 19.091 ms GCSServer.deadline_timer.debug_state_event_stats_print - 104 total (1 active, 1 running), Execution time: mean = 1.515 ms, total = 157.600 ms, Queueing time: mean = 55.827 us, max = 137.733 us, min = 13.943 us, total = 5.806 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:56:15,352 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 288156 total (1 active) Queueing time: mean = 40.499 us, max = 58.960 ms, min = 1.891 us, total = 11.670 s Execution time: mean = 215.353 us, total = 62.055 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 130977 total (0 active), Execution time: mean = 446.085 us, total = 58.427 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 130977 total (0 active), Execution time: mean = 22.748 us, total = 2.979 s, Queueing time: mean = 86.704 us, max = 58.960 ms, min = 2.005 us, total = 11.356 s event_loop_lag_probe - 24952 total (0 active), Execution time: mean = 25.527 us, total = 636.941 ms, Queueing time: mean = 8.872 us, max = 395.058 us, min = 1.891 us, total = 221.374 ms GcsTaskManager.GcJobSummary - 1249 total (1 active), Execution time: mean = 9.617 us, total = 12.012 ms, Queueing time: mean = 74.028 us, max = 3.642 ms, min = 12.167 us, total = 92.461 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:56:15,352 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30713 total (25 active) Queueing time: mean = 15.527 us, max = 600.706 us, min = 1.580 us, total = 476.881 ms Execution time: mean = 4.646 s, total = 142688.253 s Event stats: event_loop_lag_probe - 24953 total (0 active), Execution time: mean = 25.502 us, total = 636.347 ms, Queueing time: mean = 9.203 us, max = 600.706 us, min = 1.580 us, total = 229.632 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1549 total (0 active), Execution time: mean = 36.563 us, total = 56.636 ms, Queueing time: mean = 81.853 us, max = 408.208 us, min = 4.084 us, total = 126.791 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1549 total (24 active), Execution time: mean = 92.115 s, total = 142686.670 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1295 total (0 active), Execution time: mean = 102.215 us, total = 132.368 ms, Queueing time: mean = 90.362 us, max = 236.271 us, min = 11.318 us, total = 117.019 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1295 total (0 active), Execution time: mean = 549.728 us, total = 711.898 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 456.864 us, total = 9.594 ms, Queueing time: mean = 68.967 us, max = 114.456 us, min = 33.142 us, total = 1.448 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:56:15,352 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24999 total (0 active) Queueing time: mean = 9.976 us, max = 2.505 ms, min = 563.000 ns, total = 249.402 ms Execution time: mean = 26.267 us, total = 656.652 ms Event stats: event_loop_lag_probe - 24953 total (0 active), Execution time: mean = 26.168 us, total = 652.968 ms, Queueing time: mean = 9.892 us, max = 2.505 ms, min = 1.873 us, total = 246.846 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:57:15,352 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1257 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1307 - cumulative published bytes: 1041019 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:57:15,353 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 75849 total (4 active) Queueing time: mean = 66.952 us, max = 225.583 ms, min = -0.023 s, total = 5.078 s Execution time: mean = 225.322 us, total = 17.090 s Event stats: event_loop_lag_probe - 25191 total (0 active), Execution time: mean = 25.883 us, total = 652.026 ms, Queueing time: mean = 9.983 us, max = 6.895 ms, min = 1.268 us, total = 251.480 ms GcsInMemoryStore.Put - 10121 total (0 active), Execution time: mean = 27.444 us, total = 277.759 ms, Queueing time: mean = 199.793 us, max = 225.278 ms, min = 1.495 us, total = 2.022 s RayletLoadPulled - 6299 total (1 active), Execution time: mean = 243.391 us, total = 1.533 s, Queueing time: mean = 74.234 us, max = 1.351 ms, min = 10.621 us, total = 467.603 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6297 total (0 active), Execution time: mean = 75.114 us, total = 472.996 ms, Queueing time: mean = 98.526 us, max = 15.090 ms, min = 11.600 us, total = 620.419 ms NodeManagerService.grpc_client.GetResourceLoad - 6297 total (0 active), Execution time: mean = 1.345 ms, total = 8.472 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2529 total (0 active), Execution time: mean = 491.719 us, total = 1.244 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2529 total (0 active), Execution time: mean = 40.149 us, total = 101.536 ms, Queueing time: mean = 90.150 us, max = 601.063 us, min = 10.186 us, total = 227.989 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2519 total (0 active), Execution time: mean = 34.965 us, total = 88.076 ms, Queueing time: mean = 106.347 us, max = 1.496 ms, min = 14.438 us, total = 267.887 ms NodeInfoGcsService.grpc_server.CheckAlive - 2519 total (0 active), Execution time: mean = 512.826 us, total = 1.292 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2101 total (1 active), Execution time: mean = 10.404 us, total = 21.858 ms, Queueing time: mean = 72.869 us, max = 715.636 us, min = 7.259 us, total = 153.098 ms HealthCheck - 2098 total (0 active), Execution time: mean = 7.572 us, total = 15.885 ms, Queueing time: mean = 90.292 us, max = 288.247 us, min = 11.335 us, total = 189.432 ms GcsInMemoryStore.Get - 1317 total (0 active), Execution time: mean = 20.901 us, total = 27.526 ms, Queueing time: mean = 12.621 us, max = 137.899 us, min = 2.227 us, total = 16.621 ms InternalKVGcsService.grpc_server.InternalKVGet - 1316 total (0 active), Execution time: mean = 579.365 us, total = 762.444 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1316 total (0 active), Execution time: mean = 55.611 us, total = 73.184 ms, Queueing time: mean = 144.557 us, max = 22.982 ms, min = 9.184 us, total = 190.237 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1257 total (0 active), Execution time: mean = 623.810 us, total = 784.129 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1257 total (0 active), Execution time: mean = 90.349 us, total = 113.568 ms, Queueing time: mean = 105.989 us, max = 235.695 us, min = 15.511 us, total = 133.228 ms GCSServer.deadline_timer.debug_state_dump - 630 total (1 active), Execution time: mean = 1.366 ms, total = 860.499 ms, Queueing time: mean = 31.096 us, max = 178.374 us, min = -0.023 s, total = 19.590 ms GCSServer.deadline_timer.debug_state_event_stats_print - 105 total (1 active, 1 running), Execution time: mean = 1.508 ms, total = 158.370 ms, Queueing time: mean = 55.510 us, max = 137.733 us, min = 13.943 us, total = 5.829 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:57:15,353 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 290928 total (1 active) Queueing time: mean = 40.334 us, max = 58.960 ms, min = 1.891 us, total = 11.734 s Execution time: mean = 214.750 us, total = 62.477 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 132237 total (0 active), Execution time: mean = 444.812 us, total = 58.821 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 132237 total (0 active), Execution time: mean = 22.698 us, total = 3.001 s, Queueing time: mean = 86.345 us, max = 58.960 ms, min = 2.005 us, total = 11.418 s event_loop_lag_probe - 25192 total (0 active), Execution time: mean = 25.504 us, total = 642.497 ms, Queueing time: mean = 8.864 us, max = 395.058 us, min = 1.891 us, total = 223.290 ms GcsTaskManager.GcJobSummary - 1261 total (1 active), Execution time: mean = 9.606 us, total = 12.113 ms, Queueing time: mean = 73.702 us, max = 3.642 ms, min = 12.167 us, total = 92.938 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:57:15,353 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31012 total (25 active) Queueing time: mean = 15.507 us, max = 600.706 us, min = 1.580 us, total = 480.891 ms Execution time: mean = 4.700 s, total = 145748.457 s Event stats: event_loop_lag_probe - 25193 total (0 active), Execution time: mean = 25.478 us, total = 641.856 ms, Queueing time: mean = 9.195 us, max = 600.706 us, min = 1.580 us, total = 231.654 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1566 total (0 active), Execution time: mean = 36.457 us, total = 57.092 ms, Queueing time: mean = 81.679 us, max = 408.208 us, min = 4.084 us, total = 127.909 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1566 total (24 active), Execution time: mean = 93.070 s, total = 145746.862 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1307 total (0 active), Execution time: mean = 102.094 us, total = 133.437 ms, Queueing time: mean = 90.134 us, max = 236.271 us, min = 11.318 us, total = 117.805 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1307 total (0 active), Execution time: mean = 548.595 us, total = 717.014 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 450.452 us, total = 9.910 ms, Queueing time: mean = 69.626 us, max = 114.456 us, min = 33.142 us, total = 1.532 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:57:15,353 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25239 total (0 active) Queueing time: mean = 9.968 us, max = 2.505 ms, min = 563.000 ns, total = 251.575 ms Execution time: mean = 26.242 us, total = 662.310 ms Event stats: event_loop_lag_probe - 25193 total (0 active), Execution time: mean = 26.143 us, total = 658.625 ms, Queueing time: mean = 9.884 us, max = 2.505 ms, min = 1.873 us, total = 249.019 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:58:15,353 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1269 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1321 - cumulative published bytes: 1050915 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:58:15,354 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 76568 total (4 active) Queueing time: mean = 66.720 us, max = 225.583 ms, min = -0.023 s, total = 5.109 s Execution time: mean = 225.547 us, total = 17.270 s Event stats: event_loop_lag_probe - 25431 total (0 active), Execution time: mean = 25.913 us, total = 658.990 ms, Queueing time: mean = 9.990 us, max = 6.895 ms, min = 1.268 us, total = 254.046 ms GcsInMemoryStore.Put - 10217 total (0 active), Execution time: mean = 27.235 us, total = 278.260 ms, Queueing time: mean = 198.396 us, max = 225.278 ms, min = 1.495 us, total = 2.027 s RayletLoadPulled - 6359 total (1 active), Execution time: mean = 243.391 us, total = 1.548 s, Queueing time: mean = 74.208 us, max = 1.351 ms, min = 10.621 us, total = 471.890 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6357 total (0 active), Execution time: mean = 75.144 us, total = 477.692 ms, Queueing time: mean = 98.509 us, max = 15.090 ms, min = 11.600 us, total = 626.224 ms NodeManagerService.grpc_client.GetResourceLoad - 6357 total (0 active), Execution time: mean = 1.348 ms, total = 8.568 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2553 total (0 active), Execution time: mean = 492.327 us, total = 1.257 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2553 total (0 active), Execution time: mean = 40.153 us, total = 102.509 ms, Queueing time: mean = 90.306 us, max = 601.063 us, min = 10.186 us, total = 230.551 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2543 total (0 active), Execution time: mean = 35.012 us, total = 89.036 ms, Queueing time: mean = 106.573 us, max = 1.496 ms, min = 14.438 us, total = 271.016 ms NodeInfoGcsService.grpc_server.CheckAlive - 2543 total (0 active), Execution time: mean = 513.639 us, total = 1.306 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2121 total (1 active), Execution time: mean = 10.421 us, total = 22.104 ms, Queueing time: mean = 73.021 us, max = 715.636 us, min = 7.259 us, total = 154.878 ms HealthCheck - 2118 total (0 active), Execution time: mean = 7.574 us, total = 16.042 ms, Queueing time: mean = 90.492 us, max = 288.247 us, min = 11.335 us, total = 191.663 ms GcsInMemoryStore.Get - 1329 total (0 active), Execution time: mean = 20.875 us, total = 27.743 ms, Queueing time: mean = 12.616 us, max = 137.899 us, min = 2.227 us, total = 16.767 ms InternalKVGcsService.grpc_server.InternalKVGet - 1328 total (0 active), Execution time: mean = 579.712 us, total = 769.857 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1328 total (0 active), Execution time: mean = 55.629 us, total = 73.875 ms, Queueing time: mean = 144.199 us, max = 22.982 ms, min = 9.184 us, total = 191.496 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1269 total (0 active), Execution time: mean = 624.287 us, total = 792.220 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1269 total (0 active), Execution time: mean = 90.325 us, total = 114.622 ms, Queueing time: mean = 106.016 us, max = 235.695 us, min = 15.511 us, total = 134.535 ms GCSServer.deadline_timer.debug_state_dump - 636 total (1 active), Execution time: mean = 1.365 ms, total = 867.978 ms, Queueing time: mean = 31.279 us, max = 178.374 us, min = -0.023 s, total = 19.893 ms GCSServer.deadline_timer.debug_state_event_stats_print - 106 total (1 active, 1 running), Execution time: mean = 1.507 ms, total = 159.758 ms, Queueing time: mean = 55.260 us, max = 137.733 us, min = 13.943 us, total = 5.858 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:58:15,355 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 293700 total (1 active) Queueing time: mean = 40.410 us, max = 58.960 ms, min = 1.891 us, total = 11.869 s Execution time: mean = 215.088 us, total = 63.171 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 133497 total (0 active), Execution time: mean = 445.500 us, total = 59.473 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 133497 total (0 active), Execution time: mean = 22.742 us, total = 3.036 s, Queueing time: mean = 86.506 us, max = 58.960 ms, min = 2.005 us, total = 11.548 s event_loop_lag_probe - 25432 total (0 active), Execution time: mean = 25.566 us, total = 650.193 ms, Queueing time: mean = 8.890 us, max = 395.058 us, min = 1.891 us, total = 226.090 ms GcsTaskManager.GcJobSummary - 1273 total (1 active), Execution time: mean = 9.618 us, total = 12.243 ms, Queueing time: mean = 73.918 us, max = 3.642 ms, min = 12.167 us, total = 94.098 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:58:15,355 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31307 total (25 active) Queueing time: mean = 15.553 us, max = 600.706 us, min = 1.580 us, total = 486.926 ms Execution time: mean = 4.657 s, total = 145808.573 s Event stats: event_loop_lag_probe - 25432 total (0 active), Execution time: mean = 25.540 us, total = 649.522 ms, Queueing time: mean = 9.226 us, max = 600.706 us, min = 1.580 us, total = 234.646 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1580 total (0 active), Execution time: mean = 36.551 us, total = 57.751 ms, Queueing time: mean = 81.769 us, max = 408.208 us, min = 4.084 us, total = 129.196 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1580 total (24 active), Execution time: mean = 92.283 s, total = 145806.959 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1321 total (0 active), Execution time: mean = 102.076 us, total = 134.842 ms, Queueing time: mean = 90.509 us, max = 260.819 us, min = 11.318 us, total = 119.562 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1321 total (0 active), Execution time: mean = 549.244 us, total = 725.551 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 450.452 us, total = 9.910 ms, Queueing time: mean = 69.626 us, max = 114.456 us, min = 33.142 us, total = 1.532 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:58:15,355 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25479 total (0 active) Queueing time: mean = 9.993 us, max = 2.505 ms, min = 563.000 ns, total = 254.604 ms Execution time: mean = 26.288 us, total = 669.794 ms Event stats: event_loop_lag_probe - 25433 total (0 active), Execution time: mean = 26.191 us, total = 666.110 ms, Queueing time: mean = 9.910 us, max = 2.505 ms, min = 1.873 us, total = 252.048 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 06:59:15,355 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1281 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1334 - cumulative published bytes: 1060777 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 06:59:15,356 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 77287 total (4 active) Queueing time: mean = 66.447 us, max = 225.583 ms, min = -0.023 s, total = 5.135 s Execution time: mean = 225.814 us, total = 17.453 s Event stats: event_loop_lag_probe - 25671 total (0 active), Execution time: mean = 25.948 us, total = 666.122 ms, Queueing time: mean = 10.000 us, max = 6.895 ms, min = 1.268 us, total = 256.702 ms GcsInMemoryStore.Put - 10313 total (0 active), Execution time: mean = 27.027 us, total = 278.727 ms, Queueing time: mean = 196.921 us, max = 225.278 ms, min = 1.495 us, total = 2.031 s RayletLoadPulled - 6419 total (1 active), Execution time: mean = 243.625 us, total = 1.564 s, Queueing time: mean = 74.126 us, max = 1.351 ms, min = 10.621 us, total = 475.814 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6417 total (0 active), Execution time: mean = 75.187 us, total = 482.476 ms, Queueing time: mean = 98.407 us, max = 15.090 ms, min = 11.600 us, total = 631.480 ms NodeManagerService.grpc_client.GetResourceLoad - 6417 total (0 active), Execution time: mean = 1.351 ms, total = 8.667 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2577 total (0 active), Execution time: mean = 493.179 us, total = 1.271 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2577 total (0 active), Execution time: mean = 40.212 us, total = 103.627 ms, Queueing time: mean = 90.383 us, max = 601.063 us, min = 10.186 us, total = 232.918 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2567 total (0 active), Execution time: mean = 35.053 us, total = 89.982 ms, Queueing time: mean = 106.614 us, max = 1.496 ms, min = 14.438 us, total = 273.678 ms NodeInfoGcsService.grpc_server.CheckAlive - 2567 total (0 active), Execution time: mean = 513.796 us, total = 1.319 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2141 total (1 active), Execution time: mean = 10.457 us, total = 22.389 ms, Queueing time: mean = 73.029 us, max = 715.636 us, min = 7.259 us, total = 156.354 ms HealthCheck - 2138 total (0 active), Execution time: mean = 7.577 us, total = 16.200 ms, Queueing time: mean = 90.620 us, max = 288.247 us, min = 11.335 us, total = 193.746 ms GcsInMemoryStore.Get - 1341 total (0 active), Execution time: mean = 20.872 us, total = 27.989 ms, Queueing time: mean = 12.598 us, max = 137.899 us, min = 2.227 us, total = 16.893 ms InternalKVGcsService.grpc_server.InternalKVGet - 1340 total (0 active), Execution time: mean = 579.588 us, total = 776.648 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1340 total (0 active), Execution time: mean = 55.758 us, total = 74.716 ms, Queueing time: mean = 143.638 us, max = 22.982 ms, min = 9.184 us, total = 192.475 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1281 total (0 active), Execution time: mean = 624.371 us, total = 799.820 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1281 total (0 active), Execution time: mean = 90.481 us, total = 115.906 ms, Queueing time: mean = 105.901 us, max = 273.843 us, min = 15.511 us, total = 135.659 ms GCSServer.deadline_timer.debug_state_dump - 642 total (1 active), Execution time: mean = 1.364 ms, total = 875.499 ms, Queueing time: mean = 31.604 us, max = 178.374 us, min = -0.023 s, total = 20.290 ms GCSServer.deadline_timer.debug_state_event_stats_print - 107 total (1 active, 1 running), Execution time: mean = 1.508 ms, total = 161.378 ms, Queueing time: mean = 54.949 us, max = 137.733 us, min = 13.943 us, total = 5.880 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 06:59:15,356 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 296472 total (1 active) Queueing time: mean = 40.500 us, max = 58.960 ms, min = 1.891 us, total = 12.007 s Execution time: mean = 215.438 us, total = 63.871 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 134757 total (0 active), Execution time: mean = 446.215 us, total = 60.131 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 134757 total (0 active), Execution time: mean = 22.781 us, total = 3.070 s, Queueing time: mean = 86.697 us, max = 58.960 ms, min = 2.005 us, total = 11.683 s event_loop_lag_probe - 25672 total (0 active), Execution time: mean = 25.649 us, total = 658.448 ms, Queueing time: mean = 8.920 us, max = 395.058 us, min = 1.891 us, total = 228.996 ms GcsTaskManager.GcJobSummary - 1285 total (1 active), Execution time: mean = 9.656 us, total = 12.408 ms, Queueing time: mean = 74.098 us, max = 3.642 ms, min = 12.167 us, total = 95.216 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 06:59:15,356 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31599 total (25 active) Queueing time: mean = 15.601 us, max = 600.706 us, min = 1.580 us, total = 492.974 ms Execution time: mean = 4.616 s, total = 145868.809 s Event stats: event_loop_lag_probe - 25672 total (0 active), Execution time: mean = 25.614 us, total = 657.563 ms, Queueing time: mean = 9.262 us, max = 600.706 us, min = 1.580 us, total = 237.783 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1593 total (0 active), Execution time: mean = 36.577 us, total = 58.267 ms, Queueing time: mean = 82.036 us, max = 408.208 us, min = 4.084 us, total = 130.683 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1593 total (24 active), Execution time: mean = 91.568 s, total = 145867.176 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1334 total (0 active), Execution time: mean = 102.468 us, total = 136.693 ms, Queueing time: mean = 90.694 us, max = 260.819 us, min = 11.318 us, total = 120.985 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1334 total (0 active), Execution time: mean = 550.412 us, total = 734.250 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 450.452 us, total = 9.910 ms, Queueing time: mean = 69.626 us, max = 114.456 us, min = 33.142 us, total = 1.532 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 06:59:15,356 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25718 total (0 active) Queueing time: mean = 10.025 us, max = 2.505 ms, min = 563.000 ns, total = 257.810 ms Execution time: mean = 26.359 us, total = 677.900 ms Event stats: event_loop_lag_probe - 25672 total (0 active), Execution time: mean = 26.263 us, total = 674.216 ms, Queueing time: mean = 9.943 us, max = 2.505 ms, min = 1.873 us, total = 255.254 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:00:15,357 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1293 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1346 - cumulative published bytes: 1070605 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:00:15,358 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 78005 total (4 active) Queueing time: mean = 66.220 us, max = 225.583 ms, min = -0.023 s, total = 5.165 s Execution time: mean = 226.120 us, total = 17.639 s Event stats: event_loop_lag_probe - 25911 total (0 active), Execution time: mean = 25.967 us, total = 672.837 ms, Queueing time: mean = 10.011 us, max = 6.895 ms, min = 1.268 us, total = 259.389 ms GcsInMemoryStore.Put - 10409 total (0 active), Execution time: mean = 26.853 us, total = 279.518 ms, Queueing time: mean = 195.534 us, max = 225.278 ms, min = 1.495 us, total = 2.035 s RayletLoadPulled - 6479 total (1 active), Execution time: mean = 243.847 us, total = 1.580 s, Queueing time: mean = 74.028 us, max = 1.351 ms, min = 10.621 us, total = 479.627 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6477 total (0 active), Execution time: mean = 75.194 us, total = 487.034 ms, Queueing time: mean = 98.511 us, max = 15.090 ms, min = 11.600 us, total = 638.057 ms NodeManagerService.grpc_client.GetResourceLoad - 6477 total (0 active), Execution time: mean = 1.353 ms, total = 8.765 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2601 total (0 active), Execution time: mean = 493.442 us, total = 1.283 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2601 total (0 active), Execution time: mean = 40.233 us, total = 104.646 ms, Queueing time: mean = 90.424 us, max = 601.063 us, min = 10.186 us, total = 235.192 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2591 total (0 active), Execution time: mean = 35.135 us, total = 91.036 ms, Queueing time: mean = 106.703 us, max = 1.496 ms, min = 14.438 us, total = 276.468 ms NodeInfoGcsService.grpc_server.CheckAlive - 2591 total (0 active), Execution time: mean = 516.652 us, total = 1.339 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2160 total (1 active), Execution time: mean = 10.470 us, total = 22.615 ms, Queueing time: mean = 73.163 us, max = 715.636 us, min = 7.259 us, total = 158.033 ms HealthCheck - 2158 total (0 active), Execution time: mean = 7.578 us, total = 16.354 ms, Queueing time: mean = 91.111 us, max = 1.139 ms, min = 11.335 us, total = 196.619 ms GcsInMemoryStore.Get - 1353 total (0 active), Execution time: mean = 20.855 us, total = 28.217 ms, Queueing time: mean = 12.584 us, max = 137.899 us, min = 2.227 us, total = 17.026 ms InternalKVGcsService.grpc_server.InternalKVGet - 1352 total (0 active), Execution time: mean = 578.944 us, total = 782.733 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1352 total (0 active), Execution time: mean = 55.884 us, total = 75.555 ms, Queueing time: mean = 143.036 us, max = 22.982 ms, min = 9.184 us, total = 193.384 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1293 total (0 active), Execution time: mean = 624.234 us, total = 807.135 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1293 total (0 active), Execution time: mean = 90.549 us, total = 117.079 ms, Queueing time: mean = 105.803 us, max = 273.843 us, min = 15.511 us, total = 136.803 ms GCSServer.deadline_timer.debug_state_dump - 648 total (1 active), Execution time: mean = 1.363 ms, total = 883.249 ms, Queueing time: mean = 32.127 us, max = 178.374 us, min = -0.023 s, total = 20.818 ms GCSServer.deadline_timer.debug_state_event_stats_print - 108 total (1 active, 1 running), Execution time: mean = 1.508 ms, total = 162.878 ms, Queueing time: mean = 55.451 us, max = 137.733 us, min = 13.943 us, total = 5.989 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:00:15,358 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 299244 total (1 active) Queueing time: mean = 40.552 us, max = 58.960 ms, min = 1.891 us, total = 12.135 s Execution time: mean = 215.672 us, total = 64.538 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 136017 total (0 active), Execution time: mean = 446.696 us, total = 60.758 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 136017 total (0 active), Execution time: mean = 22.806 us, total = 3.102 s, Queueing time: mean = 86.808 us, max = 58.960 ms, min = 2.005 us, total = 11.807 s event_loop_lag_probe - 25912 total (0 active), Execution time: mean = 25.688 us, total = 665.633 ms, Queueing time: mean = 8.933 us, max = 395.058 us, min = 1.891 us, total = 231.464 ms GcsTaskManager.GcJobSummary - 1297 total (1 active), Execution time: mean = 9.670 us, total = 12.542 ms, Queueing time: mean = 74.086 us, max = 3.642 ms, min = 12.167 us, total = 96.089 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:00:15,358 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31887 total (25 active) Queueing time: mean = 15.621 us, max = 600.706 us, min = 1.580 us, total = 498.093 ms Execution time: mean = 4.576 s, total = 145928.983 s Event stats: event_loop_lag_probe - 25912 total (0 active), Execution time: mean = 25.640 us, total = 664.386 ms, Queueing time: mean = 9.275 us, max = 600.706 us, min = 1.580 us, total = 240.343 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1605 total (0 active), Execution time: mean = 36.642 us, total = 58.810 ms, Queueing time: mean = 82.186 us, max = 408.208 us, min = 4.084 us, total = 131.909 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1605 total (24 active), Execution time: mean = 90.920 s, total = 145927.334 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1346 total (0 active), Execution time: mean = 102.543 us, total = 138.022 ms, Queueing time: mean = 90.875 us, max = 260.819 us, min = 11.318 us, total = 122.318 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1346 total (0 active), Execution time: mean = 550.859 us, total = 741.456 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 450.452 us, total = 9.910 ms, Queueing time: mean = 69.626 us, max = 114.456 us, min = 33.142 us, total = 1.532 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:00:15,358 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25958 total (0 active) Queueing time: mean = 10.033 us, max = 2.505 ms, min = 563.000 ns, total = 260.427 ms Execution time: mean = 26.381 us, total = 684.809 ms Event stats: event_loop_lag_probe - 25912 total (0 active), Execution time: mean = 26.286 us, total = 681.125 ms, Queueing time: mean = 9.952 us, max = 2.505 ms, min = 1.873 us, total = 257.870 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:01:15,358 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1305 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1359 - cumulative published bytes: 1080467 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:01:15,360 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 78724 total (4 active) Queueing time: mean = 65.951 us, max = 225.583 ms, min = -0.023 s, total = 5.192 s Execution time: mean = 225.876 us, total = 17.782 s Event stats: event_loop_lag_probe - 26151 total (0 active), Execution time: mean = 25.939 us, total = 678.325 ms, Queueing time: mean = 9.992 us, max = 6.895 ms, min = 1.268 us, total = 261.314 ms GcsInMemoryStore.Put - 10505 total (0 active), Execution time: mean = 26.648 us, total = 279.932 ms, Queueing time: mean = 194.095 us, max = 225.278 ms, min = 1.495 us, total = 2.039 s RayletLoadPulled - 6539 total (1 active), Execution time: mean = 243.584 us, total = 1.593 s, Queueing time: mean = 74.018 us, max = 1.351 ms, min = 10.621 us, total = 484.003 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6537 total (0 active), Execution time: mean = 75.094 us, total = 490.892 ms, Queueing time: mean = 98.498 us, max = 15.090 ms, min = 11.600 us, total = 643.883 ms NodeManagerService.grpc_client.GetResourceLoad - 6537 total (0 active), Execution time: mean = 1.353 ms, total = 8.844 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2625 total (0 active), Execution time: mean = 491.719 us, total = 1.291 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2625 total (0 active), Execution time: mean = 40.167 us, total = 105.438 ms, Queueing time: mean = 90.220 us, max = 601.063 us, min = 10.186 us, total = 236.827 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2615 total (0 active), Execution time: mean = 35.064 us, total = 91.692 ms, Queueing time: mean = 106.596 us, max = 1.496 ms, min = 14.438 us, total = 278.749 ms NodeInfoGcsService.grpc_server.CheckAlive - 2615 total (0 active), Execution time: mean = 515.205 us, total = 1.347 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2180 total (1 active), Execution time: mean = 10.450 us, total = 22.780 ms, Queueing time: mean = 73.783 us, max = 1.648 ms, min = 7.259 us, total = 160.847 ms HealthCheck - 2178 total (0 active), Execution time: mean = 7.567 us, total = 16.481 ms, Queueing time: mean = 90.993 us, max = 1.139 ms, min = 11.335 us, total = 198.183 ms GcsInMemoryStore.Get - 1365 total (0 active), Execution time: mean = 22.430 us, total = 30.617 ms, Queueing time: mean = 12.562 us, max = 137.899 us, min = 2.227 us, total = 17.147 ms InternalKVGcsService.grpc_server.InternalKVGet - 1364 total (0 active), Execution time: mean = 578.524 us, total = 789.106 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1364 total (0 active), Execution time: mean = 55.887 us, total = 76.229 ms, Queueing time: mean = 142.415 us, max = 22.982 ms, min = 9.184 us, total = 194.254 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1305 total (0 active), Execution time: mean = 622.269 us, total = 812.061 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1305 total (0 active), Execution time: mean = 90.413 us, total = 117.988 ms, Queueing time: mean = 105.581 us, max = 273.843 us, min = 15.511 us, total = 137.784 ms GCSServer.deadline_timer.debug_state_dump - 654 total (1 active), Execution time: mean = 1.361 ms, total = 890.097 ms, Queueing time: mean = 32.353 us, max = 178.374 us, min = -0.023 s, total = 21.159 ms GCSServer.deadline_timer.debug_state_event_stats_print - 109 total (1 active, 1 running), Execution time: mean = 1.510 ms, total = 164.615 ms, Queueing time: mean = 55.693 us, max = 137.733 us, min = 13.943 us, total = 6.071 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:01:15,360 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 302016 total (1 active) Queueing time: mean = 40.303 us, max = 58.960 ms, min = 1.891 us, total = 12.172 s Execution time: mean = 214.767 us, total = 64.863 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 137277 total (0 active), Execution time: mean = 444.784 us, total = 61.059 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 137277 total (0 active), Execution time: mean = 22.730 us, total = 3.120 s, Queueing time: mean = 86.264 us, max = 58.960 ms, min = 2.005 us, total = 11.842 s event_loop_lag_probe - 26152 total (0 active), Execution time: mean = 25.678 us, total = 671.522 ms, Queueing time: mean = 8.930 us, max = 395.058 us, min = 1.891 us, total = 233.536 ms GcsTaskManager.GcJobSummary - 1309 total (1 active), Execution time: mean = 9.643 us, total = 12.623 ms, Queueing time: mean = 73.739 us, max = 3.642 ms, min = 12.167 us, total = 96.525 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:01:15,360 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32179 total (25 active) Queueing time: mean = 15.567 us, max = 600.706 us, min = 1.580 us, total = 500.926 ms Execution time: mean = 4.537 s, total = 145989.057 s Event stats: event_loop_lag_probe - 26152 total (0 active), Execution time: mean = 25.618 us, total = 669.974 ms, Queueing time: mean = 9.269 us, max = 600.706 us, min = 1.580 us, total = 242.404 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1618 total (0 active), Execution time: mean = 36.579 us, total = 59.186 ms, Queueing time: mean = 81.668 us, max = 408.208 us, min = 4.084 us, total = 132.139 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1618 total (24 active), Execution time: mean = 90.227 s, total = 145987.397 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1359 total (0 active), Execution time: mean = 102.356 us, total = 139.101 ms, Queueing time: mean = 90.404 us, max = 260.819 us, min = 11.318 us, total = 122.859 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1359 total (0 active), Execution time: mean = 548.721 us, total = 745.712 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 450.452 us, total = 9.910 ms, Queueing time: mean = 69.626 us, max = 114.456 us, min = 33.142 us, total = 1.532 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:01:15,360 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26198 total (0 active) Queueing time: mean = 10.020 us, max = 2.505 ms, min = 563.000 ns, total = 262.500 ms Execution time: mean = 26.351 us, total = 690.332 ms Event stats: event_loop_lag_probe - 26152 total (0 active), Execution time: mean = 26.256 us, total = 686.648 ms, Queueing time: mean = 9.940 us, max = 2.505 ms, min = 1.873 us, total = 259.944 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:02:15,360 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1317 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1372 - cumulative published bytes: 1090329 - current buffered bytes: 153 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:02:15,361 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 79443 total (4 active) Queueing time: mean = 65.713 us, max = 225.583 ms, min = -0.023 s, total = 5.220 s Execution time: mean = 225.913 us, total = 17.947 s Event stats: event_loop_lag_probe - 26391 total (0 active), Execution time: mean = 25.936 us, total = 684.473 ms, Queueing time: mean = 9.983 us, max = 6.895 ms, min = 1.268 us, total = 263.471 ms GcsInMemoryStore.Put - 10601 total (0 active), Execution time: mean = 26.455 us, total = 280.452 ms, Queueing time: mean = 192.697 us, max = 225.278 ms, min = 1.495 us, total = 2.043 s RayletLoadPulled - 6599 total (1 active), Execution time: mean = 243.543 us, total = 1.607 s, Queueing time: mean = 74.067 us, max = 1.351 ms, min = 10.621 us, total = 488.768 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6597 total (0 active), Execution time: mean = 75.080 us, total = 495.304 ms, Queueing time: mean = 98.574 us, max = 15.090 ms, min = 11.600 us, total = 650.295 ms NodeManagerService.grpc_client.GetResourceLoad - 6597 total (0 active), Execution time: mean = 1.354 ms, total = 8.931 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2649 total (0 active), Execution time: mean = 492.078 us, total = 1.304 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2649 total (0 active), Execution time: mean = 40.203 us, total = 106.499 ms, Queueing time: mean = 90.211 us, max = 601.063 us, min = 10.186 us, total = 238.970 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2639 total (0 active), Execution time: mean = 35.047 us, total = 92.488 ms, Queueing time: mean = 106.670 us, max = 1.496 ms, min = 14.438 us, total = 281.503 ms NodeInfoGcsService.grpc_server.CheckAlive - 2639 total (0 active), Execution time: mean = 515.391 us, total = 1.360 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2200 total (1 active), Execution time: mean = 10.440 us, total = 22.968 ms, Queueing time: mean = 74.030 us, max = 1.648 ms, min = 7.259 us, total = 162.865 ms HealthCheck - 2198 total (0 active), Execution time: mean = 7.568 us, total = 16.634 ms, Queueing time: mean = 90.919 us, max = 1.139 ms, min = 11.335 us, total = 199.840 ms GcsInMemoryStore.Get - 1377 total (0 active), Execution time: mean = 22.397 us, total = 30.840 ms, Queueing time: mean = 12.551 us, max = 137.899 us, min = 2.227 us, total = 17.283 ms InternalKVGcsService.grpc_server.InternalKVGet - 1376 total (0 active), Execution time: mean = 578.227 us, total = 795.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1376 total (0 active), Execution time: mean = 55.845 us, total = 76.843 ms, Queueing time: mean = 141.896 us, max = 22.982 ms, min = 9.184 us, total = 195.249 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1317 total (0 active), Execution time: mean = 622.237 us, total = 819.487 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1317 total (0 active), Execution time: mean = 90.298 us, total = 118.922 ms, Queueing time: mean = 105.517 us, max = 273.843 us, min = 15.511 us, total = 138.966 ms GCSServer.deadline_timer.debug_state_dump - 660 total (1 active), Execution time: mean = 1.360 ms, total = 897.638 ms, Queueing time: mean = 32.682 us, max = 178.374 us, min = -0.023 s, total = 21.570 ms GCSServer.deadline_timer.debug_state_event_stats_print - 110 total (1 active, 1 running), Execution time: mean = 1.512 ms, total = 166.355 ms, Queueing time: mean = 55.791 us, max = 137.733 us, min = 13.943 us, total = 6.137 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:02:15,361 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 304787 total (1 active) Queueing time: mean = 40.308 us, max = 58.960 ms, min = 1.891 us, total = 12.285 s Execution time: mean = 214.768 us, total = 65.459 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 138537 total (0 active), Execution time: mean = 444.772 us, total = 61.617 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 138537 total (0 active), Execution time: mean = 22.736 us, total = 3.150 s, Queueing time: mean = 86.272 us, max = 58.960 ms, min = 2.005 us, total = 11.952 s event_loop_lag_probe - 26391 total (0 active), Execution time: mean = 25.718 us, total = 678.721 ms, Queueing time: mean = 8.942 us, max = 395.058 us, min = 1.891 us, total = 236.002 ms GcsTaskManager.GcJobSummary - 1321 total (1 active), Execution time: mean = 9.630 us, total = 12.721 ms, Queueing time: mean = 73.927 us, max = 3.642 ms, min = 12.167 us, total = 97.657 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:02:15,362 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32508 total (25 active) Queueing time: mean = 15.602 us, max = 600.706 us, min = 1.580 us, total = 507.178 ms Execution time: mean = 4.825 s, total = 156849.191 s Event stats: event_loop_lag_probe - 26392 total (0 active), Execution time: mean = 25.632 us, total = 676.481 ms, Queueing time: mean = 9.275 us, max = 600.706 us, min = 1.580 us, total = 244.794 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1649 total (0 active), Execution time: mean = 36.300 us, total = 59.858 ms, Queueing time: mean = 81.671 us, max = 408.208 us, min = 4.084 us, total = 134.675 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1649 total (24 active), Execution time: mean = 95.117 s, total = 156847.515 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1372 total (0 active), Execution time: mean = 102.355 us, total = 140.431 ms, Queueing time: mean = 90.477 us, max = 260.819 us, min = 11.318 us, total = 124.134 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1372 total (0 active), Execution time: mean = 548.754 us, total = 752.890 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 448.002 us, total = 10.304 ms, Queueing time: mean = 68.833 us, max = 114.456 us, min = 33.142 us, total = 1.583 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:02:15,362 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26438 total (0 active) Queueing time: mean = 10.016 us, max = 2.505 ms, min = 563.000 ns, total = 264.801 ms Execution time: mean = 26.343 us, total = 696.469 ms Event stats: event_loop_lag_probe - 26392 total (0 active), Execution time: mean = 26.250 us, total = 692.785 ms, Queueing time: mean = 9.937 us, max = 2.505 ms, min = 1.873 us, total = 262.245 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:03:15,362 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1329 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1384 - cumulative published bytes: 1100157 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:03:15,363 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 80162 total (4 active) Queueing time: mean = 65.474 us, max = 225.583 ms, min = -0.023 s, total = 5.249 s Execution time: mean = 225.979 us, total = 18.115 s Event stats: event_loop_lag_probe - 26631 total (0 active), Execution time: mean = 25.938 us, total = 690.755 ms, Queueing time: mean = 9.976 us, max = 6.895 ms, min = 1.268 us, total = 265.670 ms GcsInMemoryStore.Put - 10697 total (0 active), Execution time: mean = 26.268 us, total = 280.984 ms, Queueing time: mean = 191.338 us, max = 225.278 ms, min = 1.495 us, total = 2.047 s RayletLoadPulled - 6659 total (1 active), Execution time: mean = 243.507 us, total = 1.622 s, Queueing time: mean = 74.142 us, max = 1.351 ms, min = 10.621 us, total = 493.710 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6657 total (0 active), Execution time: mean = 75.098 us, total = 499.929 ms, Queueing time: mean = 98.520 us, max = 15.090 ms, min = 11.600 us, total = 655.845 ms NodeManagerService.grpc_client.GetResourceLoad - 6657 total (0 active), Execution time: mean = 1.355 ms, total = 9.020 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2673 total (0 active), Execution time: mean = 492.073 us, total = 1.315 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2673 total (0 active), Execution time: mean = 40.190 us, total = 107.428 ms, Queueing time: mean = 90.253 us, max = 601.063 us, min = 10.186 us, total = 241.246 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2663 total (0 active), Execution time: mean = 35.058 us, total = 93.361 ms, Queueing time: mean = 106.652 us, max = 1.496 ms, min = 14.438 us, total = 284.014 ms NodeInfoGcsService.grpc_server.CheckAlive - 2663 total (0 active), Execution time: mean = 515.705 us, total = 1.373 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2220 total (1 active), Execution time: mean = 10.430 us, total = 23.154 ms, Queueing time: mean = 73.974 us, max = 1.648 ms, min = 7.259 us, total = 164.222 ms HealthCheck - 2218 total (0 active), Execution time: mean = 7.572 us, total = 16.795 ms, Queueing time: mean = 91.041 us, max = 1.139 ms, min = 11.335 us, total = 201.929 ms GcsInMemoryStore.Get - 1389 total (0 active), Execution time: mean = 22.375 us, total = 31.079 ms, Queueing time: mean = 12.551 us, max = 137.899 us, min = 2.227 us, total = 17.434 ms InternalKVGcsService.grpc_server.InternalKVGet - 1388 total (0 active), Execution time: mean = 577.654 us, total = 801.784 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1388 total (0 active), Execution time: mean = 55.854 us, total = 77.526 ms, Queueing time: mean = 141.538 us, max = 22.982 ms, min = 9.184 us, total = 196.455 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1329 total (0 active), Execution time: mean = 622.487 us, total = 827.285 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1329 total (0 active), Execution time: mean = 90.493 us, total = 120.266 ms, Queueing time: mean = 105.575 us, max = 273.843 us, min = 15.511 us, total = 140.310 ms GCSServer.deadline_timer.debug_state_dump - 666 total (1 active), Execution time: mean = 1.360 ms, total = 905.647 ms, Queueing time: mean = 33.049 us, max = 178.374 us, min = -0.023 s, total = 22.011 ms GCSServer.deadline_timer.debug_state_event_stats_print - 111 total (1 active, 1 running), Execution time: mean = 1.513 ms, total = 167.980 ms, Queueing time: mean = 55.819 us, max = 137.733 us, min = 13.943 us, total = 6.196 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:03:15,363 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 307559 total (1 active) Queueing time: mean = 40.326 us, max = 58.960 ms, min = 1.891 us, total = 12.403 s Execution time: mean = 214.855 us, total = 66.081 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 139797 total (0 active), Execution time: mean = 444.950 us, total = 62.203 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 139797 total (0 active), Execution time: mean = 22.744 us, total = 3.180 s, Queueing time: mean = 86.305 us, max = 58.960 ms, min = 2.005 us, total = 12.065 s event_loop_lag_probe - 26631 total (0 active), Execution time: mean = 25.739 us, total = 685.455 ms, Queueing time: mean = 8.949 us, max = 395.058 us, min = 1.891 us, total = 238.315 ms GcsTaskManager.GcJobSummary - 1333 total (1 active), Execution time: mean = 9.619 us, total = 12.822 ms, Queueing time: mean = 74.278 us, max = 3.642 ms, min = 12.167 us, total = 99.012 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:03:15,363 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32796 total (25 active) Queueing time: mean = 15.607 us, max = 600.706 us, min = 1.580 us, total = 511.843 ms Execution time: mean = 4.784 s, total = 156909.377 s Event stats: event_loop_lag_probe - 26632 total (0 active), Execution time: mean = 25.638 us, total = 682.794 ms, Queueing time: mean = 9.272 us, max = 600.706 us, min = 1.580 us, total = 246.944 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1661 total (0 active), Execution time: mean = 36.383 us, total = 60.432 ms, Queueing time: mean = 81.815 us, max = 408.208 us, min = 4.084 us, total = 135.894 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1661 total (24 active), Execution time: mean = 94.466 s, total = 156907.685 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1384 total (0 active), Execution time: mean = 102.509 us, total = 141.872 ms, Queueing time: mean = 90.629 us, max = 260.819 us, min = 11.318 us, total = 125.430 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1384 total (0 active), Execution time: mean = 549.427 us, total = 760.407 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 448.002 us, total = 10.304 ms, Queueing time: mean = 68.833 us, max = 114.456 us, min = 33.142 us, total = 1.583 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:03:15,364 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26678 total (0 active) Queueing time: mean = 10.010 us, max = 2.505 ms, min = 563.000 ns, total = 267.059 ms Execution time: mean = 26.339 us, total = 702.683 ms Event stats: event_loop_lag_probe - 26632 total (0 active), Execution time: mean = 26.247 us, total = 698.998 ms, Queueing time: mean = 9.932 us, max = 2.505 ms, min = 1.873 us, total = 264.503 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:04:15,364 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1341 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1397 - cumulative published bytes: 1110019 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:04:15,365 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 80881 total (4 active) Queueing time: mean = 65.223 us, max = 225.583 ms, min = -0.023 s, total = 5.275 s Execution time: mean = 225.992 us, total = 18.278 s Event stats: event_loop_lag_probe - 26871 total (0 active), Execution time: mean = 25.929 us, total = 696.726 ms, Queueing time: mean = 9.968 us, max = 6.895 ms, min = 1.268 us, total = 267.854 ms GcsInMemoryStore.Put - 10793 total (0 active), Execution time: mean = 26.077 us, total = 281.448 ms, Queueing time: mean = 189.980 us, max = 225.278 ms, min = 1.495 us, total = 2.050 s RayletLoadPulled - 6719 total (1 active), Execution time: mean = 243.487 us, total = 1.636 s, Queueing time: mean = 74.134 us, max = 1.351 ms, min = 10.621 us, total = 498.107 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6717 total (0 active), Execution time: mean = 75.080 us, total = 504.312 ms, Queueing time: mean = 98.457 us, max = 15.090 ms, min = 11.600 us, total = 661.337 ms NodeManagerService.grpc_client.GetResourceLoad - 6717 total (0 active), Execution time: mean = 1.356 ms, total = 9.106 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2697 total (0 active), Execution time: mean = 492.121 us, total = 1.327 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2697 total (0 active), Execution time: mean = 40.177 us, total = 108.357 ms, Queueing time: mean = 90.286 us, max = 601.063 us, min = 10.186 us, total = 243.502 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2687 total (0 active), Execution time: mean = 35.041 us, total = 94.155 ms, Queueing time: mean = 106.645 us, max = 1.496 ms, min = 14.438 us, total = 286.556 ms NodeInfoGcsService.grpc_server.CheckAlive - 2687 total (0 active), Execution time: mean = 515.743 us, total = 1.386 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2240 total (1 active), Execution time: mean = 10.412 us, total = 23.324 ms, Queueing time: mean = 74.019 us, max = 1.648 ms, min = 7.259 us, total = 165.802 ms HealthCheck - 2238 total (0 active), Execution time: mean = 7.578 us, total = 16.958 ms, Queueing time: mean = 91.102 us, max = 1.139 ms, min = 11.335 us, total = 203.887 ms GcsInMemoryStore.Get - 1401 total (0 active), Execution time: mean = 22.337 us, total = 31.294 ms, Queueing time: mean = 12.539 us, max = 137.899 us, min = 2.227 us, total = 17.567 ms InternalKVGcsService.grpc_server.InternalKVGet - 1400 total (0 active), Execution time: mean = 577.202 us, total = 808.083 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1400 total (0 active), Execution time: mean = 55.886 us, total = 78.240 ms, Queueing time: mean = 140.988 us, max = 22.982 ms, min = 9.184 us, total = 197.384 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1341 total (0 active), Execution time: mean = 622.319 us, total = 834.530 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1341 total (0 active), Execution time: mean = 90.629 us, total = 121.533 ms, Queueing time: mean = 105.433 us, max = 273.843 us, min = 15.511 us, total = 141.386 ms GCSServer.deadline_timer.debug_state_dump - 672 total (1 active), Execution time: mean = 1.360 ms, total = 913.862 ms, Queueing time: mean = 33.328 us, max = 178.374 us, min = -0.023 s, total = 22.396 ms GCSServer.deadline_timer.debug_state_event_stats_print - 112 total (1 active, 1 running), Execution time: mean = 1.516 ms, total = 169.763 ms, Queueing time: mean = 56.093 us, max = 137.733 us, min = 13.943 us, total = 6.282 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:04:15,365 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 310331 total (1 active) Queueing time: mean = 40.326 us, max = 58.960 ms, min = 1.891 us, total = 12.514 s Execution time: mean = 214.874 us, total = 66.682 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 141057 total (0 active), Execution time: mean = 444.972 us, total = 62.766 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 141057 total (0 active), Execution time: mean = 22.759 us, total = 3.210 s, Queueing time: mean = 86.301 us, max = 58.960 ms, min = 2.005 us, total = 12.173 s event_loop_lag_probe - 26871 total (0 active), Execution time: mean = 25.772 us, total = 692.510 ms, Queueing time: mean = 8.960 us, max = 395.058 us, min = 1.891 us, total = 240.757 ms GcsTaskManager.GcJobSummary - 1345 total (1 active), Execution time: mean = 9.616 us, total = 12.933 ms, Queueing time: mean = 74.577 us, max = 3.642 ms, min = 12.167 us, total = 100.307 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:04:15,365 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33088 total (25 active) Queueing time: mean = 15.585 us, max = 600.706 us, min = 1.580 us, total = 515.672 ms Execution time: mean = 4.744 s, total = 156969.511 s Event stats: event_loop_lag_probe - 26872 total (0 active), Execution time: mean = 25.632 us, total = 688.790 ms, Queueing time: mean = 9.266 us, max = 600.706 us, min = 1.580 us, total = 249.005 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1674 total (0 active), Execution time: mean = 36.413 us, total = 60.956 ms, Queueing time: mean = 81.663 us, max = 408.208 us, min = 4.084 us, total = 136.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1674 total (24 active), Execution time: mean = 93.768 s, total = 156967.804 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1397 total (0 active), Execution time: mean = 102.550 us, total = 143.262 ms, Queueing time: mean = 90.471 us, max = 260.819 us, min = 11.318 us, total = 126.388 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1397 total (0 active), Execution time: mean = 549.429 us, total = 767.553 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 448.002 us, total = 10.304 ms, Queueing time: mean = 68.833 us, max = 114.456 us, min = 33.142 us, total = 1.583 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:04:15,365 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26918 total (0 active) Queueing time: mean = 10.004 us, max = 2.505 ms, min = 563.000 ns, total = 269.283 ms Execution time: mean = 26.323 us, total = 708.559 ms Event stats: event_loop_lag_probe - 26872 total (0 active), Execution time: mean = 26.231 us, total = 704.874 ms, Queueing time: mean = 9.926 us, max = 2.505 ms, min = 1.873 us, total = 266.727 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:05:15,365 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1353 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1409 - cumulative published bytes: 1119847 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:05:15,366 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 81600 total (5 active) Queueing time: mean = 65.008 us, max = 225.583 ms, min = -0.023 s, total = 5.305 s Execution time: mean = 226.108 us, total = 18.450 s Event stats: event_loop_lag_probe - 27111 total (1 active), Execution time: mean = 25.927 us, total = 702.894 ms, Queueing time: mean = 9.962 us, max = 6.895 ms, min = 1.268 us, total = 270.066 ms GcsInMemoryStore.Put - 10889 total (0 active), Execution time: mean = 25.891 us, total = 281.930 ms, Queueing time: mean = 188.707 us, max = 225.278 ms, min = 1.495 us, total = 2.055 s RayletLoadPulled - 6779 total (1 active), Execution time: mean = 243.421 us, total = 1.650 s, Queueing time: mean = 74.142 us, max = 1.351 ms, min = 10.621 us, total = 502.610 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6777 total (0 active), Execution time: mean = 75.048 us, total = 508.600 ms, Queueing time: mean = 98.493 us, max = 15.090 ms, min = 11.600 us, total = 667.484 ms NodeManagerService.grpc_client.GetResourceLoad - 6777 total (0 active), Execution time: mean = 1.357 ms, total = 9.197 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2721 total (0 active), Execution time: mean = 492.625 us, total = 1.340 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2721 total (0 active), Execution time: mean = 40.241 us, total = 109.497 ms, Queueing time: mean = 90.317 us, max = 601.063 us, min = 10.186 us, total = 245.752 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2711 total (0 active), Execution time: mean = 35.066 us, total = 95.064 ms, Queueing time: mean = 106.753 us, max = 1.496 ms, min = 14.438 us, total = 289.407 ms NodeInfoGcsService.grpc_server.CheckAlive - 2711 total (0 active), Execution time: mean = 516.302 us, total = 1.400 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2260 total (1 active), Execution time: mean = 10.393 us, total = 23.489 ms, Queueing time: mean = 74.045 us, max = 1.648 ms, min = 7.259 us, total = 167.343 ms HealthCheck - 2258 total (0 active), Execution time: mean = 7.583 us, total = 17.123 ms, Queueing time: mean = 91.070 us, max = 1.139 ms, min = 11.335 us, total = 205.635 ms GcsInMemoryStore.Get - 1413 total (0 active), Execution time: mean = 22.333 us, total = 31.557 ms, Queueing time: mean = 12.536 us, max = 137.899 us, min = 2.227 us, total = 17.713 ms InternalKVGcsService.grpc_server.InternalKVGet - 1412 total (0 active), Execution time: mean = 577.152 us, total = 814.938 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1412 total (0 active), Execution time: mean = 55.910 us, total = 78.945 ms, Queueing time: mean = 140.673 us, max = 22.982 ms, min = 9.184 us, total = 198.630 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1353 total (0 active), Execution time: mean = 623.153 us, total = 843.127 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1353 total (0 active), Execution time: mean = 91.028 us, total = 123.161 ms, Queueing time: mean = 105.590 us, max = 273.843 us, min = 15.511 us, total = 142.863 ms GCSServer.deadline_timer.debug_state_dump - 678 total (1 active), Execution time: mean = 1.359 ms, total = 921.612 ms, Queueing time: mean = 33.936 us, max = 178.374 us, min = -0.023 s, total = 23.008 ms GCSServer.deadline_timer.debug_state_event_stats_print - 113 total (1 active, 1 running), Execution time: mean = 1.513 ms, total = 171.017 ms, Queueing time: mean = 57.711 us, max = 238.837 us, min = 13.943 us, total = 6.521 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:05:15,366 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 313103 total (1 active) Queueing time: mean = 40.394 us, max = 58.960 ms, min = 1.891 us, total = 12.648 s Execution time: mean = 215.127 us, total = 67.357 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 142317 total (0 active), Execution time: mean = 445.487 us, total = 63.400 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 142317 total (0 active), Execution time: mean = 22.793 us, total = 3.244 s, Queueing time: mean = 86.447 us, max = 58.960 ms, min = 2.005 us, total = 12.303 s event_loop_lag_probe - 27111 total (0 active), Execution time: mean = 25.803 us, total = 699.545 ms, Queueing time: mean = 8.973 us, max = 395.058 us, min = 1.891 us, total = 243.264 ms GcsTaskManager.GcJobSummary - 1357 total (1 active), Execution time: mean = 9.613 us, total = 13.045 ms, Queueing time: mean = 74.818 us, max = 3.642 ms, min = 12.167 us, total = 101.528 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:05:15,367 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33376 total (25 active) Queueing time: mean = 15.597 us, max = 600.706 us, min = 1.580 us, total = 520.555 ms Execution time: mean = 4.705 s, total = 157029.717 s Event stats: event_loop_lag_probe - 27112 total (0 active), Execution time: mean = 25.643 us, total = 695.232 ms, Queueing time: mean = 9.269 us, max = 600.706 us, min = 1.580 us, total = 251.305 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1686 total (0 active), Execution time: mean = 36.453 us, total = 61.459 ms, Queueing time: mean = 81.880 us, max = 408.208 us, min = 4.084 us, total = 138.050 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1686 total (24 active), Execution time: mean = 93.136 s, total = 157027.995 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1409 total (0 active), Execution time: mean = 102.556 us, total = 144.502 ms, Queueing time: mean = 90.579 us, max = 260.819 us, min = 11.318 us, total = 127.626 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1409 total (0 active), Execution time: mean = 549.781 us, total = 774.641 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 448.002 us, total = 10.304 ms, Queueing time: mean = 68.833 us, max = 114.456 us, min = 33.142 us, total = 1.583 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:05:15,367 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27158 total (0 active) Queueing time: mean = 10.002 us, max = 2.505 ms, min = 563.000 ns, total = 271.642 ms Execution time: mean = 26.315 us, total = 714.675 ms Event stats: event_loop_lag_probe - 27112 total (0 active), Execution time: mean = 26.224 us, total = 710.991 ms, Queueing time: mean = 9.925 us, max = 2.505 ms, min = 1.873 us, total = 269.086 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:06:15,367 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1365 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1421 - cumulative published bytes: 1129675 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:06:15,368 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 82318 total (4 active) Queueing time: mean = 64.786 us, max = 225.583 ms, min = -0.023 s, total = 5.333 s Execution time: mean = 225.747 us, total = 18.583 s Event stats: event_loop_lag_probe - 27350 total (0 active), Execution time: mean = 25.911 us, total = 708.664 ms, Queueing time: mean = 10.010 us, max = 6.895 ms, min = 1.268 us, total = 273.763 ms GcsInMemoryStore.Put - 10985 total (0 active), Execution time: mean = 25.704 us, total = 282.362 ms, Queueing time: mean = 187.422 us, max = 225.278 ms, min = 1.495 us, total = 2.059 s RayletLoadPulled - 6839 total (1 active), Execution time: mean = 243.222 us, total = 1.663 s, Queueing time: mean = 74.144 us, max = 1.351 ms, min = 10.621 us, total = 507.068 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6837 total (0 active), Execution time: mean = 75.006 us, total = 512.813 ms, Queueing time: mean = 98.496 us, max = 15.090 ms, min = 11.600 us, total = 673.419 ms NodeManagerService.grpc_client.GetResourceLoad - 6837 total (0 active), Execution time: mean = 1.355 ms, total = 9.263 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2745 total (0 active), Execution time: mean = 491.567 us, total = 1.349 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2745 total (0 active), Execution time: mean = 40.221 us, total = 110.408 ms, Queueing time: mean = 90.228 us, max = 601.063 us, min = 10.186 us, total = 247.676 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2735 total (0 active), Execution time: mean = 35.089 us, total = 95.969 ms, Queueing time: mean = 106.748 us, max = 1.496 ms, min = 14.438 us, total = 291.955 ms NodeInfoGcsService.grpc_server.CheckAlive - 2735 total (0 active), Execution time: mean = 515.482 us, total = 1.410 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2280 total (1 active), Execution time: mean = 10.381 us, total = 23.668 ms, Queueing time: mean = 74.033 us, max = 1.648 ms, min = 7.259 us, total = 168.794 ms HealthCheck - 2278 total (0 active), Execution time: mean = 7.583 us, total = 17.274 ms, Queueing time: mean = 91.022 us, max = 1.139 ms, min = 11.335 us, total = 207.349 ms GcsInMemoryStore.Get - 1425 total (0 active), Execution time: mean = 22.278 us, total = 31.746 ms, Queueing time: mean = 12.523 us, max = 137.899 us, min = 2.227 us, total = 17.845 ms InternalKVGcsService.grpc_server.InternalKVGet - 1424 total (0 active), Execution time: mean = 575.380 us, total = 819.342 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1424 total (0 active), Execution time: mean = 55.888 us, total = 79.584 ms, Queueing time: mean = 140.136 us, max = 22.982 ms, min = 9.184 us, total = 199.554 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1365 total (0 active), Execution time: mean = 621.784 us, total = 848.735 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1365 total (0 active), Execution time: mean = 90.944 us, total = 124.139 ms, Queueing time: mean = 105.528 us, max = 273.843 us, min = 15.511 us, total = 144.046 ms GCSServer.deadline_timer.debug_state_dump - 684 total (1 active), Execution time: mean = 1.359 ms, total = 929.841 ms, Queueing time: mean = 34.154 us, max = 178.374 us, min = -0.023 s, total = 23.361 ms GCSServer.deadline_timer.debug_state_event_stats_print - 114 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 172.593 ms, Queueing time: mean = 57.884 us, max = 238.837 us, min = 13.943 us, total = 6.599 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:06:15,368 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 315875 total (1 active) Queueing time: mean = 40.205 us, max = 58.960 ms, min = 1.891 us, total = 12.700 s Execution time: mean = 214.445 us, total = 67.738 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 143577 total (0 active), Execution time: mean = 444.037 us, total = 63.754 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 143577 total (0 active), Execution time: mean = 22.742 us, total = 3.265 s, Queueing time: mean = 86.032 us, max = 58.960 ms, min = 2.005 us, total = 12.352 s event_loop_lag_probe - 27351 total (0 active), Execution time: mean = 25.808 us, total = 705.874 ms, Queueing time: mean = 8.976 us, max = 395.058 us, min = 1.891 us, total = 245.494 ms GcsTaskManager.GcJobSummary - 1369 total (1 active), Execution time: mean = 9.605 us, total = 13.150 ms, Queueing time: mean = 74.495 us, max = 3.642 ms, min = 12.167 us, total = 101.983 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:06:15,368 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33664 total (25 active) Queueing time: mean = 15.556 us, max = 600.706 us, min = 1.580 us, total = 523.689 ms Execution time: mean = 4.666 s, total = 157089.915 s Event stats: event_loop_lag_probe - 27352 total (0 active), Execution time: mean = 25.628 us, total = 700.988 ms, Queueing time: mean = 9.265 us, max = 600.706 us, min = 1.580 us, total = 253.404 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1698 total (0 active), Execution time: mean = 36.423 us, total = 61.846 ms, Queueing time: mean = 81.585 us, max = 408.208 us, min = 4.084 us, total = 138.532 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1698 total (24 active), Execution time: mean = 92.514 s, total = 157088.181 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1421 total (0 active), Execution time: mean = 102.473 us, total = 145.614 ms, Queueing time: mean = 90.203 us, max = 260.819 us, min = 11.318 us, total = 128.178 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1421 total (0 active), Execution time: mean = 548.442 us, total = 779.336 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 448.002 us, total = 10.304 ms, Queueing time: mean = 68.833 us, max = 114.456 us, min = 33.142 us, total = 1.583 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:06:15,368 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27398 total (0 active) Queueing time: mean = 9.993 us, max = 2.505 ms, min = 563.000 ns, total = 273.788 ms Execution time: mean = 26.291 us, total = 720.319 ms Event stats: event_loop_lag_probe - 27352 total (0 active), Execution time: mean = 26.200 us, total = 716.635 ms, Queueing time: mean = 9.916 us, max = 2.505 ms, min = 1.873 us, total = 271.232 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:07:15,369 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1377 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1436 - cumulative published bytes: 1139605 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:07:15,370 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 83037 total (4 active) Queueing time: mean = 64.501 us, max = 225.583 ms, min = -0.023 s, total = 5.356 s Execution time: mean = 225.203 us, total = 18.700 s Event stats: event_loop_lag_probe - 27590 total (0 active), Execution time: mean = 25.891 us, total = 714.331 ms, Queueing time: mean = 9.997 us, max = 6.895 ms, min = 1.268 us, total = 275.821 ms GcsInMemoryStore.Put - 11081 total (0 active), Execution time: mean = 25.517 us, total = 282.750 ms, Queueing time: mean = 186.225 us, max = 225.278 ms, min = 1.495 us, total = 2.064 s RayletLoadPulled - 6899 total (1 active), Execution time: mean = 242.937 us, total = 1.676 s, Queueing time: mean = 74.092 us, max = 1.351 ms, min = 10.621 us, total = 511.162 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6897 total (0 active), Execution time: mean = 74.912 us, total = 516.666 ms, Queueing time: mean = 98.248 us, max = 15.090 ms, min = 11.600 us, total = 677.619 ms NodeManagerService.grpc_client.GetResourceLoad - 6897 total (0 active), Execution time: mean = 1.351 ms, total = 9.321 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2769 total (0 active), Execution time: mean = 489.742 us, total = 1.356 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2769 total (0 active), Execution time: mean = 40.186 us, total = 111.274 ms, Queueing time: mean = 89.906 us, max = 601.063 us, min = 10.186 us, total = 248.950 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2759 total (0 active), Execution time: mean = 35.007 us, total = 96.585 ms, Queueing time: mean = 106.559 us, max = 1.496 ms, min = 14.438 us, total = 293.997 ms NodeInfoGcsService.grpc_server.CheckAlive - 2759 total (0 active), Execution time: mean = 513.880 us, total = 1.418 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2300 total (1 active), Execution time: mean = 10.357 us, total = 23.821 ms, Queueing time: mean = 73.968 us, max = 1.648 ms, min = 7.259 us, total = 170.127 ms HealthCheck - 2298 total (0 active), Execution time: mean = 7.566 us, total = 17.388 ms, Queueing time: mean = 90.685 us, max = 1.139 ms, min = 11.335 us, total = 208.394 ms GcsInMemoryStore.Get - 1437 total (0 active), Execution time: mean = 22.218 us, total = 31.927 ms, Queueing time: mean = 12.495 us, max = 137.899 us, min = 2.227 us, total = 17.956 ms InternalKVGcsService.grpc_server.InternalKVGet - 1436 total (0 active), Execution time: mean = 573.115 us, total = 822.992 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1436 total (0 active), Execution time: mean = 55.772 us, total = 80.088 ms, Queueing time: mean = 139.448 us, max = 22.982 ms, min = 9.184 us, total = 200.248 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1377 total (0 active), Execution time: mean = 619.737 us, total = 853.378 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1377 total (0 active), Execution time: mean = 90.962 us, total = 125.255 ms, Queueing time: mean = 105.236 us, max = 273.843 us, min = 15.511 us, total = 144.910 ms GCSServer.deadline_timer.debug_state_dump - 690 total (1 active), Execution time: mean = 1.360 ms, total = 938.211 ms, Queueing time: mean = 34.457 us, max = 178.374 us, min = -0.023 s, total = 23.776 ms GCSServer.deadline_timer.debug_state_event_stats_print - 115 total (1 active, 1 running), Execution time: mean = 1.515 ms, total = 174.176 ms, Queueing time: mean = 57.845 us, max = 238.837 us, min = 13.943 us, total = 6.652 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:07:15,370 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 318647 total (1 active) Queueing time: mean = 39.947 us, max = 58.960 ms, min = 1.891 us, total = 12.729 s Execution time: mean = 213.493 us, total = 68.029 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 144837 total (0 active), Execution time: mean = 442.034 us, total = 64.023 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 144837 total (0 active), Execution time: mean = 22.657 us, total = 3.282 s, Queueing time: mean = 85.469 us, max = 58.960 ms, min = 2.005 us, total = 12.379 s event_loop_lag_probe - 27591 total (0 active), Execution time: mean = 25.779 us, total = 711.278 ms, Queueing time: mean = 8.967 us, max = 395.058 us, min = 1.891 us, total = 247.413 ms GcsTaskManager.GcJobSummary - 1381 total (1 active), Execution time: mean = 9.576 us, total = 13.225 ms, Queueing time: mean = 74.115 us, max = 3.642 ms, min = 12.167 us, total = 102.353 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:07:15,370 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33975 total (25 active) Queueing time: mean = 15.492 us, max = 600.706 us, min = 1.580 us, total = 526.324 ms Execution time: mean = 4.714 s, total = 160150.004 s Event stats: event_loop_lag_probe - 27592 total (0 active), Execution time: mean = 25.595 us, total = 706.212 ms, Queueing time: mean = 9.251 us, max = 600.706 us, min = 1.580 us, total = 255.246 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1718 total (0 active), Execution time: mean = 36.244 us, total = 62.267 ms, Queueing time: mean = 80.857 us, max = 408.208 us, min = 4.084 us, total = 138.912 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1718 total (24 active), Execution time: mean = 93.218 s, total = 160148.259 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1436 total (0 active), Execution time: mean = 102.196 us, total = 146.754 ms, Queueing time: mean = 89.518 us, max = 260.819 us, min = 11.318 us, total = 128.547 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1436 total (0 active), Execution time: mean = 545.572 us, total = 783.442 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 436.037 us, total = 10.465 ms, Queueing time: mean = 67.821 us, max = 114.456 us, min = 33.142 us, total = 1.628 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:07:15,370 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27638 total (0 active) Queueing time: mean = 9.974 us, max = 2.505 ms, min = 563.000 ns, total = 275.656 ms Execution time: mean = 26.243 us, total = 725.307 ms Event stats: event_loop_lag_probe - 27592 total (0 active), Execution time: mean = 26.153 us, total = 721.623 ms, Queueing time: mean = 9.898 us, max = 2.505 ms, min = 1.873 us, total = 273.100 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:08:15,370 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1389 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1449 - cumulative published bytes: 1149467 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:08:15,371 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 83756 total (4 active) Queueing time: mean = 64.251 us, max = 225.583 ms, min = -0.023 s, total = 5.381 s Execution time: mean = 224.625 us, total = 18.814 s Event stats: event_loop_lag_probe - 27830 total (0 active), Execution time: mean = 25.866 us, total = 719.838 ms, Queueing time: mean = 9.985 us, max = 6.895 ms, min = 1.268 us, total = 277.891 ms GcsInMemoryStore.Put - 11177 total (0 active), Execution time: mean = 25.336 us, total = 283.181 ms, Queueing time: mean = 184.978 us, max = 225.278 ms, min = 1.495 us, total = 2.067 s RayletLoadPulled - 6959 total (1 active), Execution time: mean = 242.660 us, total = 1.689 s, Queueing time: mean = 74.011 us, max = 1.351 ms, min = 10.621 us, total = 515.043 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6957 total (0 active), Execution time: mean = 74.833 us, total = 520.610 ms, Queueing time: mean = 98.130 us, max = 15.090 ms, min = 11.600 us, total = 682.693 ms NodeManagerService.grpc_client.GetResourceLoad - 6957 total (0 active), Execution time: mean = 1.348 ms, total = 9.376 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2793 total (0 active), Execution time: mean = 488.002 us, total = 1.363 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2793 total (0 active), Execution time: mean = 40.066 us, total = 111.905 ms, Queueing time: mean = 89.756 us, max = 601.063 us, min = 10.186 us, total = 250.687 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2783 total (0 active), Execution time: mean = 34.968 us, total = 97.317 ms, Queueing time: mean = 106.460 us, max = 1.496 ms, min = 14.438 us, total = 296.278 ms NodeInfoGcsService.grpc_server.CheckAlive - 2783 total (0 active), Execution time: mean = 512.368 us, total = 1.426 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2320 total (1 active), Execution time: mean = 10.332 us, total = 23.970 ms, Queueing time: mean = 74.145 us, max = 1.648 ms, min = 7.259 us, total = 172.017 ms HealthCheck - 2318 total (0 active), Execution time: mean = 7.555 us, total = 17.513 ms, Queueing time: mean = 90.743 us, max = 1.139 ms, min = 11.335 us, total = 210.343 ms GcsInMemoryStore.Get - 1449 total (0 active), Execution time: mean = 22.152 us, total = 32.098 ms, Queueing time: mean = 12.472 us, max = 137.899 us, min = 2.227 us, total = 18.072 ms InternalKVGcsService.grpc_server.InternalKVGet - 1448 total (0 active), Execution time: mean = 570.954 us, total = 826.742 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1448 total (0 active), Execution time: mean = 55.632 us, total = 80.556 ms, Queueing time: mean = 138.974 us, max = 22.982 ms, min = 9.184 us, total = 201.235 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1389 total (0 active), Execution time: mean = 617.693 us, total = 857.975 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1389 total (0 active), Execution time: mean = 90.843 us, total = 126.181 ms, Queueing time: mean = 105.086 us, max = 273.843 us, min = 15.511 us, total = 145.964 ms GCSServer.deadline_timer.debug_state_dump - 696 total (1 active), Execution time: mean = 1.359 ms, total = 946.076 ms, Queueing time: mean = 34.745 us, max = 178.374 us, min = -0.023 s, total = 24.183 ms GCSServer.deadline_timer.debug_state_event_stats_print - 116 total (1 active, 1 running), Execution time: mean = 1.517 ms, total = 175.942 ms, Queueing time: mean = 58.017 us, max = 238.837 us, min = 13.943 us, total = 6.730 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:08:15,371 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 321415 total (1 active) Queueing time: mean = 39.690 us, max = 58.960 ms, min = 1.891 us, total = 12.757 s Execution time: mean = 212.545 us, total = 68.315 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 146095 total (0 active), Execution time: mean = 440.037 us, total = 64.287 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 146095 total (0 active), Execution time: mean = 22.574 us, total = 3.298 s, Queueing time: mean = 84.909 us, max = 58.960 ms, min = 2.005 us, total = 12.405 s event_loop_lag_probe - 27831 total (0 active), Execution time: mean = 25.753 us, total = 716.718 ms, Queueing time: mean = 8.961 us, max = 395.058 us, min = 1.891 us, total = 249.388 ms GcsTaskManager.GcJobSummary - 1393 total (1 active), Execution time: mean = 9.552 us, total = 13.306 ms, Queueing time: mean = 73.733 us, max = 3.642 ms, min = 12.167 us, total = 102.710 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:08:15,371 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34267 total (25 active) Queueing time: mean = 15.430 us, max = 600.706 us, min = 1.580 us, total = 528.726 ms Execution time: mean = 4.675 s, total = 160210.163 s Event stats: event_loop_lag_probe - 27832 total (0 active), Execution time: mean = 25.559 us, total = 711.359 ms, Queueing time: mean = 9.237 us, max = 600.706 us, min = 1.580 us, total = 257.094 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1731 total (0 active), Execution time: mean = 36.183 us, total = 62.633 ms, Queueing time: mean = 80.376 us, max = 408.208 us, min = 4.084 us, total = 139.131 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1731 total (24 active), Execution time: mean = 92.553 s, total = 160208.408 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1449 total (0 active), Execution time: mean = 101.918 us, total = 147.680 ms, Queueing time: mean = 88.946 us, max = 260.819 us, min = 11.233 us, total = 128.882 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1449 total (0 active), Execution time: mean = 543.200 us, total = 787.097 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 436.037 us, total = 10.465 ms, Queueing time: mean = 67.821 us, max = 114.456 us, min = 33.142 us, total = 1.628 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:08:15,371 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27878 total (0 active) Queueing time: mean = 9.957 us, max = 2.505 ms, min = 563.000 ns, total = 277.594 ms Execution time: mean = 26.197 us, total = 730.315 ms Event stats: event_loop_lag_probe - 27832 total (0 active), Execution time: mean = 26.108 us, total = 726.631 ms, Queueing time: mean = 9.882 us, max = 2.505 ms, min = 1.873 us, total = 275.038 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:09:15,372 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1401 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1461 - cumulative published bytes: 1159295 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:09:15,373 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 84472 total (4 active) Queueing time: mean = 64.005 us, max = 225.583 ms, min = -0.023 s, total = 5.407 s Execution time: mean = 224.072 us, total = 18.928 s Event stats: event_loop_lag_probe - 28070 total (0 active), Execution time: mean = 25.853 us, total = 725.700 ms, Queueing time: mean = 9.978 us, max = 6.895 ms, min = 1.268 us, total = 280.077 ms GcsInMemoryStore.Put - 11273 total (0 active), Execution time: mean = 25.161 us, total = 283.636 ms, Queueing time: mean = 183.726 us, max = 225.278 ms, min = 1.495 us, total = 2.071 s RayletLoadPulled - 7018 total (1 active), Execution time: mean = 242.476 us, total = 1.702 s, Queueing time: mean = 73.930 us, max = 1.351 ms, min = 10.621 us, total = 518.842 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7016 total (0 active), Execution time: mean = 74.767 us, total = 524.562 ms, Queueing time: mean = 98.102 us, max = 15.090 ms, min = 11.600 us, total = 688.284 ms NodeManagerService.grpc_client.GetResourceLoad - 7016 total (0 active), Execution time: mean = 1.344 ms, total = 9.429 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2817 total (0 active), Execution time: mean = 486.548 us, total = 1.371 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2817 total (0 active), Execution time: mean = 40.000 us, total = 112.681 ms, Queueing time: mean = 89.736 us, max = 601.063 us, min = 10.186 us, total = 252.787 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2807 total (0 active), Execution time: mean = 34.905 us, total = 97.977 ms, Queueing time: mean = 106.335 us, max = 1.496 ms, min = 14.438 us, total = 298.481 ms NodeInfoGcsService.grpc_server.CheckAlive - 2807 total (0 active), Execution time: mean = 511.312 us, total = 1.435 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2340 total (1 active), Execution time: mean = 10.309 us, total = 24.122 ms, Queueing time: mean = 73.996 us, max = 1.648 ms, min = 7.259 us, total = 173.151 ms HealthCheck - 2338 total (0 active), Execution time: mean = 7.548 us, total = 17.647 ms, Queueing time: mean = 90.697 us, max = 1.139 ms, min = 11.335 us, total = 212.050 ms GcsInMemoryStore.Get - 1461 total (0 active), Execution time: mean = 22.125 us, total = 32.325 ms, Queueing time: mean = 12.465 us, max = 137.899 us, min = 2.227 us, total = 18.212 ms InternalKVGcsService.grpc_server.InternalKVGet - 1460 total (0 active), Execution time: mean = 569.055 us, total = 830.820 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1460 total (0 active), Execution time: mean = 55.549 us, total = 81.101 ms, Queueing time: mean = 138.541 us, max = 22.982 ms, min = 9.184 us, total = 202.269 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1401 total (0 active), Execution time: mean = 616.047 us, total = 863.082 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1401 total (0 active), Execution time: mean = 90.705 us, total = 127.078 ms, Queueing time: mean = 105.149 us, max = 273.843 us, min = 15.511 us, total = 147.314 ms GCSServer.deadline_timer.debug_state_dump - 702 total (1 active), Execution time: mean = 1.357 ms, total = 952.331 ms, Queueing time: mean = 34.866 us, max = 178.374 us, min = -0.023 s, total = 24.476 ms GCSServer.deadline_timer.debug_state_event_stats_print - 117 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 177.127 ms, Queueing time: mean = 57.855 us, max = 238.837 us, min = 13.943 us, total = 6.769 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:09:15,373 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 324187 total (1 active) Queueing time: mean = 39.440 us, max = 58.960 ms, min = 1.891 us, total = 12.786 s Execution time: mean = 211.627 us, total = 68.607 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 147355 total (0 active), Execution time: mean = 438.108 us, total = 64.557 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 147355 total (0 active), Execution time: mean = 22.489 us, total = 3.314 s, Queueing time: mean = 84.363 us, max = 58.960 ms, min = 2.005 us, total = 12.431 s event_loop_lag_probe - 28071 total (0 active), Execution time: mean = 25.722 us, total = 722.035 ms, Queueing time: mean = 8.952 us, max = 395.058 us, min = 1.891 us, total = 251.295 ms GcsTaskManager.GcJobSummary - 1405 total (1 active), Execution time: mean = 9.530 us, total = 13.390 ms, Queueing time: mean = 73.413 us, max = 3.642 ms, min = 12.167 us, total = 103.145 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:09:15,373 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34555 total (25 active) Queueing time: mean = 15.370 us, max = 600.706 us, min = 1.580 us, total = 531.106 ms Execution time: mean = 4.638 s, total = 160270.302 s Event stats: event_loop_lag_probe - 28072 total (0 active), Execution time: mean = 25.524 us, total = 716.503 ms, Queueing time: mean = 9.225 us, max = 600.706 us, min = 1.580 us, total = 258.955 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1743 total (0 active), Execution time: mean = 36.103 us, total = 62.928 ms, Queueing time: mean = 79.936 us, max = 408.208 us, min = 4.084 us, total = 139.328 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1743 total (24 active), Execution time: mean = 91.950 s, total = 160268.536 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1461 total (0 active), Execution time: mean = 101.689 us, total = 148.568 ms, Queueing time: mean = 88.435 us, max = 260.819 us, min = 11.233 us, total = 129.204 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1461 total (0 active), Execution time: mean = 541.213 us, total = 790.712 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 436.037 us, total = 10.465 ms, Queueing time: mean = 67.821 us, max = 114.456 us, min = 33.142 us, total = 1.628 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:09:15,373 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28118 total (0 active) Queueing time: mean = 9.938 us, max = 2.505 ms, min = 563.000 ns, total = 279.428 ms Execution time: mean = 26.142 us, total = 735.054 ms Event stats: event_loop_lag_probe - 28072 total (0 active), Execution time: mean = 26.053 us, total = 731.370 ms, Queueing time: mean = 9.863 us, max = 2.505 ms, min = 1.873 us, total = 276.872 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:10:15,373 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1413 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1473 - cumulative published bytes: 1169123 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:10:15,374 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 85191 total (4 active) Queueing time: mean = 63.735 us, max = 225.583 ms, min = -0.023 s, total = 5.430 s Execution time: mean = 223.796 us, total = 19.065 s Event stats: event_loop_lag_probe - 28310 total (0 active), Execution time: mean = 25.846 us, total = 731.708 ms, Queueing time: mean = 9.973 us, max = 6.895 ms, min = 1.268 us, total = 282.325 ms GcsInMemoryStore.Put - 11369 total (0 active), Execution time: mean = 24.984 us, total = 284.042 ms, Queueing time: mean = 182.482 us, max = 225.278 ms, min = 1.495 us, total = 2.075 s RayletLoadPulled - 7078 total (1 active), Execution time: mean = 242.274 us, total = 1.715 s, Queueing time: mean = 73.866 us, max = 1.351 ms, min = 10.621 us, total = 522.822 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7076 total (0 active), Execution time: mean = 74.698 us, total = 528.564 ms, Queueing time: mean = 97.870 us, max = 15.090 ms, min = 11.600 us, total = 692.525 ms NodeManagerService.grpc_client.GetResourceLoad - 7076 total (0 active), Execution time: mean = 1.343 ms, total = 9.501 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2841 total (0 active), Execution time: mean = 485.600 us, total = 1.380 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2841 total (0 active), Execution time: mean = 39.886 us, total = 113.315 ms, Queueing time: mean = 89.597 us, max = 601.063 us, min = 10.186 us, total = 254.544 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2831 total (0 active), Execution time: mean = 34.856 us, total = 98.677 ms, Queueing time: mean = 106.155 us, max = 1.496 ms, min = 14.438 us, total = 300.525 ms NodeInfoGcsService.grpc_server.CheckAlive - 2831 total (0 active), Execution time: mean = 510.688 us, total = 1.446 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2360 total (1 active), Execution time: mean = 10.299 us, total = 24.306 ms, Queueing time: mean = 74.060 us, max = 1.648 ms, min = 7.259 us, total = 174.782 ms HealthCheck - 2358 total (0 active), Execution time: mean = 7.538 us, total = 17.775 ms, Queueing time: mean = 90.493 us, max = 1.139 ms, min = 11.335 us, total = 213.382 ms GcsInMemoryStore.Get - 1473 total (0 active), Execution time: mean = 22.077 us, total = 32.520 ms, Queueing time: mean = 12.446 us, max = 137.899 us, min = 2.227 us, total = 18.333 ms InternalKVGcsService.grpc_server.InternalKVGet - 1472 total (0 active), Execution time: mean = 567.978 us, total = 836.063 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1472 total (0 active), Execution time: mean = 55.470 us, total = 81.652 ms, Queueing time: mean = 137.990 us, max = 22.982 ms, min = 9.184 us, total = 203.121 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1413 total (0 active), Execution time: mean = 615.065 us, total = 869.087 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1413 total (0 active), Execution time: mean = 90.533 us, total = 127.923 ms, Queueing time: mean = 104.946 us, max = 273.843 us, min = 15.511 us, total = 148.289 ms GCSServer.deadline_timer.debug_state_dump - 708 total (1 active), Execution time: mean = 1.354 ms, total = 958.901 ms, Queueing time: mean = 35.042 us, max = 178.374 us, min = -0.023 s, total = 24.810 ms GCSServer.deadline_timer.debug_state_event_stats_print - 118 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 178.693 ms, Queueing time: mean = 57.868 us, max = 238.837 us, min = 13.943 us, total = 6.828 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:10:15,374 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 326959 total (1 active) Queueing time: mean = 39.387 us, max = 58.960 ms, min = 1.891 us, total = 12.878 s Execution time: mean = 211.414 us, total = 69.124 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 148615 total (0 active), Execution time: mean = 437.666 us, total = 65.044 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 148615 total (0 active), Execution time: mean = 22.463 us, total = 3.338 s, Queueing time: mean = 84.248 us, max = 58.960 ms, min = 2.005 us, total = 12.521 s event_loop_lag_probe - 28311 total (0 active), Execution time: mean = 25.722 us, total = 728.207 ms, Queueing time: mean = 8.952 us, max = 395.058 us, min = 1.891 us, total = 253.440 ms GcsTaskManager.GcJobSummary - 1417 total (1 active), Execution time: mean = 9.513 us, total = 13.481 ms, Queueing time: mean = 73.308 us, max = 3.642 ms, min = 12.167 us, total = 103.878 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:10:15,374 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34842 total (25 active) Queueing time: mean = 15.345 us, max = 600.706 us, min = 1.580 us, total = 534.664 ms Execution time: mean = 4.602 s, total = 160330.412 s Event stats: event_loop_lag_probe - 28311 total (0 active), Execution time: mean = 25.511 us, total = 722.256 ms, Queueing time: mean = 9.219 us, max = 600.706 us, min = 1.580 us, total = 260.992 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1755 total (0 active), Execution time: mean = 36.071 us, total = 63.304 ms, Queueing time: mean = 79.754 us, max = 408.208 us, min = 4.084 us, total = 139.968 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1755 total (24 active), Execution time: mean = 91.355 s, total = 160328.634 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1473 total (0 active), Execution time: mean = 101.602 us, total = 149.659 ms, Queueing time: mean = 88.313 us, max = 260.819 us, min = 11.233 us, total = 130.085 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1473 total (0 active), Execution time: mean = 540.549 us, total = 796.229 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 436.037 us, total = 10.465 ms, Queueing time: mean = 67.821 us, max = 114.456 us, min = 33.142 us, total = 1.628 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:10:15,375 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28358 total (0 active) Queueing time: mean = 9.924 us, max = 2.505 ms, min = 563.000 ns, total = 281.435 ms Execution time: mean = 26.107 us, total = 740.330 ms Event stats: event_loop_lag_probe - 28312 total (0 active), Execution time: mean = 26.019 us, total = 736.645 ms, Queueing time: mean = 9.850 us, max = 2.505 ms, min = 1.873 us, total = 278.879 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:11:15,375 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1425 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1487 - cumulative published bytes: 1179019 - current buffered bytes: 153 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:11:15,376 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 85910 total (4 active) Queueing time: mean = 63.548 us, max = 225.583 ms, min = -0.023 s, total = 5.459 s Execution time: mean = 223.970 us, total = 19.241 s Event stats: event_loop_lag_probe - 28550 total (0 active), Execution time: mean = 25.865 us, total = 738.452 ms, Queueing time: mean = 9.977 us, max = 6.895 ms, min = 1.268 us, total = 284.841 ms GcsInMemoryStore.Put - 11465 total (0 active), Execution time: mean = 24.823 us, total = 284.594 ms, Queueing time: mean = 181.330 us, max = 225.278 ms, min = 1.495 us, total = 2.079 s RayletLoadPulled - 7138 total (1 active), Execution time: mean = 242.461 us, total = 1.731 s, Queueing time: mean = 73.918 us, max = 1.351 ms, min = 10.621 us, total = 527.626 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7136 total (0 active), Execution time: mean = 74.687 us, total = 532.966 ms, Queueing time: mean = 97.874 us, max = 15.090 ms, min = 11.600 us, total = 698.426 ms NodeManagerService.grpc_client.GetResourceLoad - 7136 total (0 active), Execution time: mean = 1.345 ms, total = 9.595 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2865 total (0 active), Execution time: mean = 486.356 us, total = 1.393 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2865 total (0 active), Execution time: mean = 39.939 us, total = 114.425 ms, Queueing time: mean = 89.696 us, max = 601.063 us, min = 10.186 us, total = 256.978 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2855 total (0 active), Execution time: mean = 34.911 us, total = 99.672 ms, Queueing time: mean = 106.136 us, max = 1.496 ms, min = 14.438 us, total = 303.018 ms NodeInfoGcsService.grpc_server.CheckAlive - 2855 total (0 active), Execution time: mean = 511.057 us, total = 1.459 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2380 total (1 active), Execution time: mean = 10.298 us, total = 24.510 ms, Queueing time: mean = 74.209 us, max = 1.648 ms, min = 7.259 us, total = 176.617 ms HealthCheck - 2378 total (0 active), Execution time: mean = 7.545 us, total = 17.941 ms, Queueing time: mean = 90.585 us, max = 1.139 ms, min = 11.335 us, total = 215.411 ms GcsInMemoryStore.Get - 1485 total (0 active), Execution time: mean = 22.067 us, total = 32.770 ms, Queueing time: mean = 12.444 us, max = 137.899 us, min = 2.227 us, total = 18.479 ms InternalKVGcsService.grpc_server.InternalKVGet - 1484 total (0 active), Execution time: mean = 568.246 us, total = 843.277 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1484 total (0 active), Execution time: mean = 55.486 us, total = 82.342 ms, Queueing time: mean = 137.829 us, max = 22.982 ms, min = 9.184 us, total = 204.538 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1425 total (0 active), Execution time: mean = 615.298 us, total = 876.799 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1425 total (0 active), Execution time: mean = 90.483 us, total = 128.939 ms, Queueing time: mean = 105.000 us, max = 273.843 us, min = 15.511 us, total = 149.624 ms GCSServer.deadline_timer.debug_state_dump - 714 total (1 active), Execution time: mean = 1.353 ms, total = 965.971 ms, Queueing time: mean = 35.326 us, max = 178.374 us, min = -0.023 s, total = 25.222 ms GCSServer.deadline_timer.debug_state_event_stats_print - 119 total (1 active, 1 running), Execution time: mean = 1.514 ms, total = 180.195 ms, Queueing time: mean = 57.945 us, max = 238.837 us, min = 13.943 us, total = 6.895 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:11:15,376 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 329731 total (1 active) Queueing time: mean = 39.488 us, max = 58.960 ms, min = 1.891 us, total = 13.020 s Execution time: mean = 211.786 us, total = 69.832 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 149875 total (0 active), Execution time: mean = 438.452 us, total = 65.713 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 149875 total (0 active), Execution time: mean = 22.490 us, total = 3.371 s, Queueing time: mean = 84.470 us, max = 58.960 ms, min = 2.005 us, total = 12.660 s event_loop_lag_probe - 28551 total (0 active), Execution time: mean = 25.748 us, total = 735.140 ms, Queueing time: mean = 8.960 us, max = 395.058 us, min = 1.891 us, total = 255.825 ms GcsTaskManager.GcJobSummary - 1429 total (1 active), Execution time: mean = 9.509 us, total = 13.588 ms, Queueing time: mean = 73.224 us, max = 3.642 ms, min = 12.167 us, total = 104.638 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:11:15,376 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35138 total (25 active) Queueing time: mean = 15.372 us, max = 600.706 us, min = 1.580 us, total = 540.158 ms Execution time: mean = 4.565 s, total = 160390.546 s Event stats: event_loop_lag_probe - 28551 total (0 active), Execution time: mean = 25.528 us, total = 728.840 ms, Queueing time: mean = 9.224 us, max = 600.706 us, min = 1.580 us, total = 263.342 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1769 total (0 active), Execution time: mean = 36.071 us, total = 63.810 ms, Queueing time: mean = 80.058 us, max = 408.208 us, min = 4.084 us, total = 141.623 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1769 total (24 active), Execution time: mean = 90.666 s, total = 160388.752 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1487 total (0 active), Execution time: mean = 101.484 us, total = 150.907 ms, Queueing time: mean = 88.482 us, max = 260.819 us, min = 11.233 us, total = 131.573 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1487 total (0 active), Execution time: mean = 540.863 us, total = 804.264 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 436.037 us, total = 10.465 ms, Queueing time: mean = 67.821 us, max = 114.456 us, min = 33.142 us, total = 1.628 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:11:15,376 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28598 total (0 active) Queueing time: mean = 9.918 us, max = 2.505 ms, min = 563.000 ns, total = 283.644 ms Execution time: mean = 26.091 us, total = 746.154 ms Event stats: event_loop_lag_probe - 28552 total (0 active), Execution time: mean = 26.004 us, total = 742.470 ms, Queueing time: mean = 9.845 us, max = 2.505 ms, min = 1.873 us, total = 281.088 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:12:15,377 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1437 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1499 - cumulative published bytes: 1188847 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:12:15,377 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 86629 total (4 active) Queueing time: mean = 63.376 us, max = 225.583 ms, min = -0.023 s, total = 5.490 s Execution time: mean = 224.005 us, total = 19.405 s Event stats: event_loop_lag_probe - 28790 total (0 active), Execution time: mean = 25.878 us, total = 745.027 ms, Queueing time: mean = 9.977 us, max = 6.895 ms, min = 1.268 us, total = 287.247 ms GcsInMemoryStore.Put - 11561 total (0 active), Execution time: mean = 24.663 us, total = 285.124 ms, Queueing time: mean = 180.177 us, max = 225.278 ms, min = 1.495 us, total = 2.083 s RayletLoadPulled - 7198 total (1 active), Execution time: mean = 242.507 us, total = 1.746 s, Queueing time: mean = 74.052 us, max = 1.351 ms, min = 10.621 us, total = 533.029 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7196 total (0 active), Execution time: mean = 74.755 us, total = 537.937 ms, Queueing time: mean = 98.067 us, max = 15.090 ms, min = 11.600 us, total = 705.687 ms NodeManagerService.grpc_client.GetResourceLoad - 7196 total (0 active), Execution time: mean = 1.345 ms, total = 9.681 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2889 total (0 active), Execution time: mean = 486.320 us, total = 1.405 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2889 total (0 active), Execution time: mean = 40.007 us, total = 115.581 ms, Queueing time: mean = 89.711 us, max = 601.063 us, min = 10.186 us, total = 259.175 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2879 total (0 active), Execution time: mean = 34.882 us, total = 100.427 ms, Queueing time: mean = 106.161 us, max = 1.496 ms, min = 14.438 us, total = 305.637 ms NodeInfoGcsService.grpc_server.CheckAlive - 2879 total (0 active), Execution time: mean = 511.020 us, total = 1.471 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2400 total (1 active), Execution time: mean = 10.296 us, total = 24.710 ms, Queueing time: mean = 74.388 us, max = 1.648 ms, min = 7.259 us, total = 178.531 ms HealthCheck - 2398 total (0 active), Execution time: mean = 7.551 us, total = 18.108 ms, Queueing time: mean = 90.651 us, max = 1.139 ms, min = 11.335 us, total = 217.380 ms GcsInMemoryStore.Get - 1497 total (0 active), Execution time: mean = 22.051 us, total = 33.010 ms, Queueing time: mean = 12.441 us, max = 137.899 us, min = 2.227 us, total = 18.624 ms InternalKVGcsService.grpc_server.InternalKVGet - 1496 total (0 active), Execution time: mean = 567.602 us, total = 849.133 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1496 total (0 active), Execution time: mean = 55.448 us, total = 82.951 ms, Queueing time: mean = 137.492 us, max = 22.982 ms, min = 9.184 us, total = 205.688 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1437 total (0 active), Execution time: mean = 615.297 us, total = 884.181 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1437 total (0 active), Execution time: mean = 90.635 us, total = 130.243 ms, Queueing time: mean = 104.966 us, max = 273.843 us, min = 15.511 us, total = 150.836 ms GCSServer.deadline_timer.debug_state_dump - 720 total (1 active), Execution time: mean = 1.352 ms, total = 973.579 ms, Queueing time: mean = 35.610 us, max = 178.374 us, min = -0.023 s, total = 25.640 ms GCSServer.deadline_timer.debug_state_event_stats_print - 120 total (1 active, 1 running), Execution time: mean = 1.516 ms, total = 181.866 ms, Queueing time: mean = 58.101 us, max = 238.837 us, min = 13.943 us, total = 6.972 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:12:15,378 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 332503 total (1 active) Queueing time: mean = 39.477 us, max = 58.960 ms, min = 1.891 us, total = 13.126 s Execution time: mean = 211.787 us, total = 70.420 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 151135 total (0 active), Execution time: mean = 438.451 us, total = 66.265 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 151135 total (0 active), Execution time: mean = 22.491 us, total = 3.399 s, Queueing time: mean = 84.445 us, max = 58.960 ms, min = 2.005 us, total = 12.763 s event_loop_lag_probe - 28791 total (0 active), Execution time: mean = 25.758 us, total = 741.607 ms, Queueing time: mean = 8.965 us, max = 395.058 us, min = 1.891 us, total = 258.117 ms GcsTaskManager.GcJobSummary - 1441 total (1 active), Execution time: mean = 9.517 us, total = 13.713 ms, Queueing time: mean = 73.160 us, max = 3.642 ms, min = 12.167 us, total = 105.423 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:12:15,378 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35463 total (25 active) Queueing time: mean = 15.399 us, max = 600.706 us, min = 1.580 us, total = 546.110 ms Execution time: mean = 4.829 s, total = 171250.796 s Event stats: event_loop_lag_probe - 28791 total (0 active), Execution time: mean = 25.530 us, total = 735.041 ms, Queueing time: mean = 9.223 us, max = 600.706 us, min = 1.580 us, total = 265.534 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1799 total (0 active), Execution time: mean = 35.961 us, total = 64.695 ms, Queueing time: mean = 80.235 us, max = 408.208 us, min = 4.084 us, total = 144.342 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1799 total (24 active), Execution time: mean = 95.191 s, total = 171248.986 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1499 total (0 active), Execution time: mean = 101.533 us, total = 152.198 ms, Queueing time: mean = 88.417 us, max = 260.819 us, min = 11.233 us, total = 132.537 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1499 total (0 active), Execution time: mean = 540.746 us, total = 810.579 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 446.974 us, total = 11.174 ms, Queueing time: mean = 68.192 us, max = 114.456 us, min = 33.142 us, total = 1.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:12:15,378 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28837 total (0 active) Queueing time: mean = 9.911 us, max = 2.505 ms, min = 563.000 ns, total = 285.794 ms Execution time: mean = 26.073 us, total = 751.881 ms Event stats: event_loop_lag_probe - 28791 total (0 active), Execution time: mean = 25.987 us, total = 748.197 ms, Queueing time: mean = 9.838 us, max = 2.505 ms, min = 1.873 us, total = 283.238 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:13:15,378 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1449 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1511 - cumulative published bytes: 1198675 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:13:15,379 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 87348 total (4 active) Queueing time: mean = 63.191 us, max = 225.583 ms, min = -0.023 s, total = 5.520 s Execution time: mean = 224.123 us, total = 19.577 s Event stats: event_loop_lag_probe - 29030 total (0 active), Execution time: mean = 25.886 us, total = 751.482 ms, Queueing time: mean = 9.975 us, max = 6.895 ms, min = 1.268 us, total = 289.560 ms GcsInMemoryStore.Put - 11657 total (0 active), Execution time: mean = 24.505 us, total = 285.654 ms, Queueing time: mean = 179.098 us, max = 225.278 ms, min = 1.495 us, total = 2.088 s RayletLoadPulled - 7258 total (1 active), Execution time: mean = 242.481 us, total = 1.760 s, Queueing time: mean = 74.068 us, max = 1.351 ms, min = 10.621 us, total = 537.583 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7256 total (0 active), Execution time: mean = 74.765 us, total = 542.494 ms, Queueing time: mean = 98.078 us, max = 15.090 ms, min = 11.600 us, total = 711.653 ms NodeManagerService.grpc_client.GetResourceLoad - 7256 total (0 active), Execution time: mean = 1.346 ms, total = 9.770 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2913 total (0 active), Execution time: mean = 486.992 us, total = 1.419 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2913 total (0 active), Execution time: mean = 40.025 us, total = 116.594 ms, Queueing time: mean = 89.775 us, max = 601.063 us, min = 10.186 us, total = 261.515 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2903 total (0 active), Execution time: mean = 34.870 us, total = 101.227 ms, Queueing time: mean = 106.230 us, max = 1.496 ms, min = 14.438 us, total = 308.386 ms NodeInfoGcsService.grpc_server.CheckAlive - 2903 total (0 active), Execution time: mean = 511.465 us, total = 1.485 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2420 total (1 active), Execution time: mean = 10.291 us, total = 24.905 ms, Queueing time: mean = 74.351 us, max = 1.648 ms, min = 7.259 us, total = 179.931 ms HealthCheck - 2418 total (0 active), Execution time: mean = 7.561 us, total = 18.284 ms, Queueing time: mean = 90.724 us, max = 1.139 ms, min = 11.335 us, total = 219.371 ms GcsInMemoryStore.Get - 1509 total (0 active), Execution time: mean = 22.035 us, total = 33.250 ms, Queueing time: mean = 12.439 us, max = 137.899 us, min = 2.227 us, total = 18.771 ms InternalKVGcsService.grpc_server.InternalKVGet - 1508 total (0 active), Execution time: mean = 568.001 us, total = 856.546 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1508 total (0 active), Execution time: mean = 55.438 us, total = 83.601 ms, Queueing time: mean = 137.297 us, max = 22.982 ms, min = 9.184 us, total = 207.044 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1449 total (0 active), Execution time: mean = 616.120 us, total = 892.758 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1449 total (0 active), Execution time: mean = 90.624 us, total = 131.314 ms, Queueing time: mean = 105.096 us, max = 273.843 us, min = 15.511 us, total = 152.285 ms GCSServer.deadline_timer.debug_state_dump - 726 total (1 active), Execution time: mean = 1.352 ms, total = 981.298 ms, Queueing time: mean = 35.749 us, max = 178.374 us, min = -0.023 s, total = 25.954 ms GCSServer.deadline_timer.debug_state_event_stats_print - 121 total (1 active, 1 running), Execution time: mean = 1.517 ms, total = 183.573 ms, Queueing time: mean = 57.976 us, max = 238.837 us, min = 13.943 us, total = 7.015 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:13:15,380 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 335274 total (1 active) Queueing time: mean = 39.547 us, max = 58.960 ms, min = 1.891 us, total = 13.259 s Execution time: mean = 212.084 us, total = 71.106 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 152395 total (0 active), Execution time: mean = 439.074 us, total = 66.913 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 152395 total (0 active), Execution time: mean = 22.517 us, total = 3.432 s, Queueing time: mean = 84.598 us, max = 58.960 ms, min = 2.005 us, total = 12.892 s event_loop_lag_probe - 29030 total (0 active), Execution time: mean = 25.772 us, total = 748.161 ms, Queueing time: mean = 8.970 us, max = 395.058 us, min = 1.891 us, total = 260.406 ms GcsTaskManager.GcJobSummary - 1453 total (1 active), Execution time: mean = 9.516 us, total = 13.827 ms, Queueing time: mean = 73.215 us, max = 3.642 ms, min = 12.167 us, total = 106.382 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:13:15,380 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35751 total (25 active) Queueing time: mean = 15.417 us, max = 600.706 us, min = 1.580 us, total = 551.171 ms Execution time: mean = 4.792 s, total = 171310.935 s Event stats: event_loop_lag_probe - 29031 total (0 active), Execution time: mean = 25.541 us, total = 741.486 ms, Queueing time: mean = 9.223 us, max = 600.706 us, min = 1.580 us, total = 267.751 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1811 total (0 active), Execution time: mean = 36.013 us, total = 65.220 ms, Queueing time: mean = 80.489 us, max = 408.208 us, min = 4.084 us, total = 145.766 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1811 total (24 active), Execution time: mean = 94.594 s, total = 171309.109 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1511 total (0 active), Execution time: mean = 101.599 us, total = 153.517 ms, Queueing time: mean = 88.656 us, max = 260.819 us, min = 11.233 us, total = 133.959 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1511 total (0 active), Execution time: mean = 541.562 us, total = 818.301 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 446.974 us, total = 11.174 ms, Queueing time: mean = 68.192 us, max = 114.456 us, min = 33.142 us, total = 1.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:13:15,380 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29077 total (0 active) Queueing time: mean = 9.900 us, max = 2.505 ms, min = 563.000 ns, total = 287.853 ms Execution time: mean = 26.051 us, total = 757.484 ms Event stats: event_loop_lag_probe - 29031 total (0 active), Execution time: mean = 25.965 us, total = 753.800 ms, Queueing time: mean = 9.827 us, max = 2.505 ms, min = 1.873 us, total = 285.297 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:14:15,380 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1460 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1522 - cumulative published bytes: 1207684 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:14:15,381 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 88056 total (4 active) Queueing time: mean = 62.991 us, max = 225.583 ms, min = -0.023 s, total = 5.547 s Execution time: mean = 224.127 us, total = 19.736 s Event stats: event_loop_lag_probe - 29270 total (0 active), Execution time: mean = 25.918 us, total = 758.626 ms, Queueing time: mean = 9.972 us, max = 6.895 ms, min = 1.268 us, total = 291.885 ms GcsInMemoryStore.Put - 11751 total (0 active), Execution time: mean = 24.349 us, total = 286.120 ms, Queueing time: mean = 178.016 us, max = 225.278 ms, min = 1.495 us, total = 2.092 s RayletLoadPulled - 7318 total (1 active), Execution time: mean = 242.542 us, total = 1.775 s, Queueing time: mean = 74.042 us, max = 1.351 ms, min = 10.621 us, total = 541.841 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7316 total (0 active), Execution time: mean = 74.822 us, total = 547.400 ms, Queueing time: mean = 98.088 us, max = 15.090 ms, min = 11.600 us, total = 717.609 ms NodeManagerService.grpc_client.GetResourceLoad - 7316 total (0 active), Execution time: mean = 1.347 ms, total = 9.854 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2935 total (0 active), Execution time: mean = 486.656 us, total = 1.428 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2935 total (0 active), Execution time: mean = 40.096 us, total = 117.683 ms, Queueing time: mean = 89.821 us, max = 601.063 us, min = 10.186 us, total = 263.624 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2927 total (0 active), Execution time: mean = 34.876 us, total = 102.081 ms, Queueing time: mean = 106.162 us, max = 1.496 ms, min = 14.438 us, total = 310.737 ms NodeInfoGcsService.grpc_server.CheckAlive - 2927 total (0 active), Execution time: mean = 510.851 us, total = 1.495 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2440 total (1 active), Execution time: mean = 10.283 us, total = 25.090 ms, Queueing time: mean = 74.316 us, max = 1.648 ms, min = 7.259 us, total = 181.331 ms HealthCheck - 2438 total (0 active), Execution time: mean = 7.557 us, total = 18.423 ms, Queueing time: mean = 90.627 us, max = 1.139 ms, min = 11.335 us, total = 220.948 ms GcsInMemoryStore.Get - 1520 total (0 active), Execution time: mean = 22.014 us, total = 33.461 ms, Queueing time: mean = 12.440 us, max = 137.899 us, min = 2.227 us, total = 18.909 ms InternalKVGcsService.grpc_server.InternalKVGet - 1519 total (0 active), Execution time: mean = 567.893 us, total = 862.630 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1519 total (0 active), Execution time: mean = 55.562 us, total = 84.399 ms, Queueing time: mean = 137.018 us, max = 22.982 ms, min = 9.184 us, total = 208.131 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1460 total (0 active), Execution time: mean = 615.818 us, total = 899.094 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1460 total (0 active), Execution time: mean = 90.679 us, total = 132.391 ms, Queueing time: mean = 105.135 us, max = 273.843 us, min = 15.511 us, total = 153.497 ms GCSServer.deadline_timer.debug_state_dump - 732 total (1 active), Execution time: mean = 1.353 ms, total = 990.320 ms, Queueing time: mean = 36.218 us, max = 178.374 us, min = -0.023 s, total = 26.511 ms GCSServer.deadline_timer.debug_state_event_stats_print - 122 total (1 active, 1 running), Execution time: mean = 1.518 ms, total = 185.241 ms, Queueing time: mean = 58.109 us, max = 238.837 us, min = 13.943 us, total = 7.089 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:14:15,381 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 338042 total (1 active) Queueing time: mean = 39.480 us, max = 58.960 ms, min = 1.891 us, total = 13.346 s Execution time: mean = 211.831 us, total = 71.608 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 153653 total (0 active), Execution time: mean = 438.531 us, total = 67.382 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 153653 total (0 active), Execution time: mean = 22.501 us, total = 3.457 s, Queueing time: mean = 84.450 us, max = 58.960 ms, min = 2.005 us, total = 12.976 s event_loop_lag_probe - 29270 total (0 active), Execution time: mean = 25.788 us, total = 754.819 ms, Queueing time: mean = 8.974 us, max = 395.058 us, min = 1.891 us, total = 262.672 ms GcsTaskManager.GcJobSummary - 1465 total (1 active), Execution time: mean = 9.524 us, total = 13.952 ms, Queueing time: mean = 73.118 us, max = 3.642 ms, min = 12.167 us, total = 107.118 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:14:15,381 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36035 total (25 active) Queueing time: mean = 15.397 us, max = 600.706 us, min = 1.580 us, total = 554.835 ms Execution time: mean = 4.756 s, total = 171366.119 s Event stats: event_loop_lag_probe - 29271 total (0 active), Execution time: mean = 25.524 us, total = 747.102 ms, Queueing time: mean = 9.215 us, max = 600.706 us, min = 1.580 us, total = 269.721 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1822 total (0 active), Execution time: mean = 36.046 us, total = 65.676 ms, Queueing time: mean = 80.464 us, max = 408.208 us, min = 4.084 us, total = 146.606 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1822 total (24 active), Execution time: mean = 94.053 s, total = 171364.280 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1522 total (0 active), Execution time: mean = 101.627 us, total = 154.677 ms, Queueing time: mean = 88.576 us, max = 260.819 us, min = 11.233 us, total = 134.813 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1522 total (0 active), Execution time: mean = 541.360 us, total = 823.950 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 446.974 us, total = 11.174 ms, Queueing time: mean = 68.192 us, max = 114.456 us, min = 33.142 us, total = 1.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:14:15,381 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29317 total (0 active) Queueing time: mean = 9.887 us, max = 2.505 ms, min = 563.000 ns, total = 289.849 ms Execution time: mean = 26.018 us, total = 762.772 ms Event stats: event_loop_lag_probe - 29271 total (0 active), Execution time: mean = 25.933 us, total = 759.088 ms, Queueing time: mean = 9.815 us, max = 2.505 ms, min = 1.873 us, total = 287.293 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:15:15,382 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1472 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1535 - cumulative published bytes: 1217546 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:15:15,383 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 88775 total (4 active) Queueing time: mean = 62.788 us, max = 225.583 ms, min = -0.023 s, total = 5.574 s Execution time: mean = 224.120 us, total = 19.896 s Event stats: event_loop_lag_probe - 29510 total (0 active), Execution time: mean = 25.895 us, total = 764.159 ms, Queueing time: mean = 9.964 us, max = 6.895 ms, min = 1.268 us, total = 294.034 ms GcsInMemoryStore.Put - 11847 total (0 active), Execution time: mean = 24.198 us, total = 286.669 ms, Queueing time: mean = 176.884 us, max = 225.278 ms, min = 1.495 us, total = 2.096 s RayletLoadPulled - 7378 total (1 active), Execution time: mean = 242.540 us, total = 1.789 s, Queueing time: mean = 73.967 us, max = 1.351 ms, min = 10.621 us, total = 545.725 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7376 total (0 active), Execution time: mean = 74.805 us, total = 551.761 ms, Queueing time: mean = 98.130 us, max = 15.090 ms, min = 11.600 us, total = 723.808 ms NodeManagerService.grpc_client.GetResourceLoad - 7376 total (0 active), Execution time: mean = 1.347 ms, total = 9.938 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2959 total (0 active), Execution time: mean = 486.863 us, total = 1.441 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2959 total (0 active), Execution time: mean = 40.043 us, total = 118.487 ms, Queueing time: mean = 89.929 us, max = 601.063 us, min = 10.186 us, total = 266.099 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2951 total (0 active), Execution time: mean = 34.867 us, total = 102.893 ms, Queueing time: mean = 106.092 us, max = 1.496 ms, min = 14.438 us, total = 313.078 ms NodeInfoGcsService.grpc_server.CheckAlive - 2951 total (0 active), Execution time: mean = 510.735 us, total = 1.507 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2460 total (1 active), Execution time: mean = 10.274 us, total = 25.275 ms, Queueing time: mean = 74.284 us, max = 1.648 ms, min = 7.259 us, total = 182.739 ms HealthCheck - 2458 total (0 active), Execution time: mean = 7.553 us, total = 18.565 ms, Queueing time: mean = 90.699 us, max = 1.139 ms, min = 11.335 us, total = 222.938 ms GcsInMemoryStore.Get - 1532 total (0 active), Execution time: mean = 22.021 us, total = 33.736 ms, Queueing time: mean = 12.439 us, max = 137.899 us, min = 2.227 us, total = 19.057 ms InternalKVGcsService.grpc_server.InternalKVGet - 1531 total (0 active), Execution time: mean = 567.931 us, total = 869.503 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1531 total (0 active), Execution time: mean = 55.490 us, total = 84.955 ms, Queueing time: mean = 136.772 us, max = 22.982 ms, min = 9.184 us, total = 209.398 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1472 total (0 active), Execution time: mean = 615.559 us, total = 906.103 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1472 total (0 active), Execution time: mean = 90.644 us, total = 133.427 ms, Queueing time: mean = 105.143 us, max = 273.843 us, min = 15.511 us, total = 154.771 ms GCSServer.deadline_timer.debug_state_dump - 738 total (1 active), Execution time: mean = 1.353 ms, total = 998.372 ms, Queueing time: mean = 36.490 us, max = 178.374 us, min = -0.023 s, total = 26.930 ms GCSServer.deadline_timer.debug_state_event_stats_print - 123 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 186.837 ms, Queueing time: mean = 58.069 us, max = 238.837 us, min = 13.943 us, total = 7.143 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:15:15,383 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 340814 total (1 active) Queueing time: mean = 39.525 us, max = 58.960 ms, min = 1.891 us, total = 13.471 s Execution time: mean = 212.000 us, total = 72.253 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 154913 total (0 active), Execution time: mean = 438.894 us, total = 67.990 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 154913 total (0 active), Execution time: mean = 22.510 us, total = 3.487 s, Queueing time: mean = 84.550 us, max = 58.960 ms, min = 2.005 us, total = 13.098 s event_loop_lag_probe - 29510 total (0 active), Execution time: mean = 25.788 us, total = 761.013 ms, Queueing time: mean = 8.974 us, max = 395.058 us, min = 1.891 us, total = 264.833 ms GcsTaskManager.GcJobSummary - 1477 total (1 active), Execution time: mean = 9.518 us, total = 14.058 ms, Queueing time: mean = 73.057 us, max = 3.642 ms, min = 12.167 us, total = 107.905 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:15:15,383 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36327 total (25 active) Queueing time: mean = 15.400 us, max = 600.706 us, min = 1.580 us, total = 559.428 ms Execution time: mean = 4.719 s, total = 171426.289 s Event stats: event_loop_lag_probe - 29511 total (0 active), Execution time: mean = 25.512 us, total = 752.893 ms, Queueing time: mean = 9.208 us, max = 600.706 us, min = 1.580 us, total = 271.749 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1835 total (0 active), Execution time: mean = 36.092 us, total = 66.229 ms, Queueing time: mean = 80.615 us, max = 408.208 us, min = 4.084 us, total = 147.929 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1835 total (24 active), Execution time: mean = 93.419 s, total = 171424.435 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1535 total (0 active), Execution time: mean = 101.550 us, total = 155.879 ms, Queueing time: mean = 88.634 us, max = 260.819 us, min = 11.233 us, total = 136.053 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1535 total (0 active), Execution time: mean = 541.578 us, total = 831.323 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 446.974 us, total = 11.174 ms, Queueing time: mean = 68.192 us, max = 114.456 us, min = 33.142 us, total = 1.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:15:15,383 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29557 total (0 active) Queueing time: mean = 9.877 us, max = 2.505 ms, min = 563.000 ns, total = 291.933 ms Execution time: mean = 26.000 us, total = 768.484 ms Event stats: event_loop_lag_probe - 29511 total (0 active), Execution time: mean = 25.916 us, total = 764.799 ms, Queueing time: mean = 9.806 us, max = 2.505 ms, min = 1.873 us, total = 289.377 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:16:15,383 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1484 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1547 - cumulative published bytes: 1227374 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:16:15,384 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 89493 total (4 active) Queueing time: mean = 62.607 us, max = 225.583 ms, min = -0.023 s, total = 5.603 s Execution time: mean = 224.248 us, total = 20.069 s Event stats: event_loop_lag_probe - 29749 total (0 active), Execution time: mean = 25.896 us, total = 770.385 ms, Queueing time: mean = 9.966 us, max = 6.895 ms, min = 1.268 us, total = 296.492 ms GcsInMemoryStore.Put - 11943 total (0 active), Execution time: mean = 24.051 us, total = 287.238 ms, Queueing time: mean = 175.829 us, max = 225.278 ms, min = 1.495 us, total = 2.100 s RayletLoadPulled - 7438 total (1 active), Execution time: mean = 242.671 us, total = 1.805 s, Queueing time: mean = 73.996 us, max = 1.351 ms, min = 10.621 us, total = 550.385 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7436 total (0 active), Execution time: mean = 74.860 us, total = 556.662 ms, Queueing time: mean = 98.178 us, max = 15.090 ms, min = 11.600 us, total = 730.050 ms NodeManagerService.grpc_client.GetResourceLoad - 7436 total (0 active), Execution time: mean = 1.349 ms, total = 10.030 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 2983 total (0 active), Execution time: mean = 487.200 us, total = 1.453 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2983 total (0 active), Execution time: mean = 40.025 us, total = 119.395 ms, Queueing time: mean = 89.923 us, max = 601.063 us, min = 10.186 us, total = 268.240 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2975 total (0 active), Execution time: mean = 34.856 us, total = 103.696 ms, Queueing time: mean = 106.077 us, max = 1.496 ms, min = 14.438 us, total = 315.580 ms NodeInfoGcsService.grpc_server.CheckAlive - 2975 total (0 active), Execution time: mean = 510.823 us, total = 1.520 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2480 total (1 active), Execution time: mean = 10.271 us, total = 25.471 ms, Queueing time: mean = 74.266 us, max = 1.648 ms, min = 7.259 us, total = 184.180 ms HealthCheck - 2478 total (0 active), Execution time: mean = 7.554 us, total = 18.720 ms, Queueing time: mean = 90.782 us, max = 1.139 ms, min = 11.335 us, total = 224.957 ms GcsInMemoryStore.Get - 1544 total (0 active), Execution time: mean = 22.044 us, total = 34.035 ms, Queueing time: mean = 12.438 us, max = 137.899 us, min = 2.227 us, total = 19.204 ms InternalKVGcsService.grpc_server.InternalKVGet - 1543 total (0 active), Execution time: mean = 567.950 us, total = 876.347 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1543 total (0 active), Execution time: mean = 55.488 us, total = 85.619 ms, Queueing time: mean = 136.496 us, max = 22.982 ms, min = 9.184 us, total = 210.613 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1484 total (0 active), Execution time: mean = 615.689 us, total = 913.683 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1484 total (0 active), Execution time: mean = 90.649 us, total = 134.524 ms, Queueing time: mean = 105.103 us, max = 273.843 us, min = 15.511 us, total = 155.973 ms GCSServer.deadline_timer.debug_state_dump - 744 total (1 active), Execution time: mean = 1.352 ms, total = 1.006 s, Queueing time: mean = 36.695 us, max = 178.374 us, min = -0.023 s, total = 27.301 ms GCSServer.deadline_timer.debug_state_event_stats_print - 124 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 188.386 ms, Queueing time: mean = 58.339 us, max = 238.837 us, min = 13.943 us, total = 7.234 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:16:15,385 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 343586 total (1 active) Queueing time: mean = 39.597 us, max = 58.960 ms, min = 1.891 us, total = 13.605 s Execution time: mean = 212.267 us, total = 72.932 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 156173 total (0 active), Execution time: mean = 439.459 us, total = 68.632 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 156173 total (0 active), Execution time: mean = 22.531 us, total = 3.519 s, Queueing time: mean = 84.708 us, max = 58.960 ms, min = 2.005 us, total = 13.229 s event_loop_lag_probe - 29750 total (0 active), Execution time: mean = 25.796 us, total = 767.431 ms, Queueing time: mean = 8.976 us, max = 395.058 us, min = 1.891 us, total = 267.050 ms GcsTaskManager.GcJobSummary - 1489 total (1 active), Execution time: mean = 9.515 us, total = 14.168 ms, Queueing time: mean = 73.082 us, max = 3.642 ms, min = 12.167 us, total = 108.819 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:16:15,385 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36615 total (25 active) Queueing time: mean = 15.407 us, max = 600.706 us, min = 1.580 us, total = 564.121 ms Execution time: mean = 4.684 s, total = 171486.461 s Event stats: event_loop_lag_probe - 29751 total (0 active), Execution time: mean = 25.519 us, total = 759.224 ms, Queueing time: mean = 9.210 us, max = 600.706 us, min = 1.580 us, total = 273.999 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1847 total (0 active), Execution time: mean = 36.166 us, total = 66.798 ms, Queueing time: mean = 80.759 us, max = 408.208 us, min = 4.084 us, total = 149.162 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1847 total (24 active), Execution time: mean = 92.845 s, total = 171484.592 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1547 total (0 active), Execution time: mean = 101.648 us, total = 157.250 ms, Queueing time: mean = 88.729 us, max = 260.819 us, min = 11.233 us, total = 137.264 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1547 total (0 active), Execution time: mean = 541.966 us, total = 838.422 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 446.974 us, total = 11.174 ms, Queueing time: mean = 68.192 us, max = 114.456 us, min = 33.142 us, total = 1.705 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:16:15,385 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29797 total (0 active) Queueing time: mean = 9.871 us, max = 2.505 ms, min = 563.000 ns, total = 294.135 ms Execution time: mean = 25.982 us, total = 774.180 ms Event stats: event_loop_lag_probe - 29751 total (0 active), Execution time: mean = 25.898 us, total = 770.496 ms, Queueing time: mean = 9.801 us, max = 2.505 ms, min = 1.873 us, total = 291.579 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:17:15,385 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1496 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1559 - cumulative published bytes: 1237202 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:17:15,386 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 90212 total (4 active) Queueing time: mean = 62.416 us, max = 225.583 ms, min = -0.023 s, total = 5.631 s Execution time: mean = 224.255 us, total = 20.231 s Event stats: event_loop_lag_probe - 29989 total (0 active), Execution time: mean = 25.882 us, total = 776.180 ms, Queueing time: mean = 9.963 us, max = 6.895 ms, min = 1.268 us, total = 298.785 ms GcsInMemoryStore.Put - 12039 total (0 active), Execution time: mean = 23.904 us, total = 287.781 ms, Queueing time: mean = 174.772 us, max = 225.278 ms, min = 1.495 us, total = 2.104 s RayletLoadPulled - 7498 total (1 active), Execution time: mean = 242.859 us, total = 1.821 s, Queueing time: mean = 73.941 us, max = 1.351 ms, min = 10.621 us, total = 554.413 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7496 total (0 active), Execution time: mean = 74.833 us, total = 560.945 ms, Queueing time: mean = 98.403 us, max = 15.090 ms, min = 11.600 us, total = 737.630 ms NodeManagerService.grpc_client.GetResourceLoad - 7496 total (0 active), Execution time: mean = 1.350 ms, total = 10.117 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3007 total (0 active), Execution time: mean = 487.093 us, total = 1.465 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3007 total (0 active), Execution time: mean = 39.969 us, total = 120.185 ms, Queueing time: mean = 89.878 us, max = 601.063 us, min = 10.186 us, total = 270.263 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2999 total (0 active), Execution time: mean = 34.820 us, total = 104.426 ms, Queueing time: mean = 105.814 us, max = 1.496 ms, min = 14.438 us, total = 317.335 ms NodeInfoGcsService.grpc_server.CheckAlive - 2999 total (0 active), Execution time: mean = 510.321 us, total = 1.530 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2500 total (1 active), Execution time: mean = 10.266 us, total = 25.664 ms, Queueing time: mean = 74.314 us, max = 1.648 ms, min = 7.259 us, total = 185.786 ms HealthCheck - 2498 total (0 active), Execution time: mean = 7.558 us, total = 18.879 ms, Queueing time: mean = 90.731 us, max = 1.139 ms, min = 11.335 us, total = 226.645 ms GcsInMemoryStore.Get - 1556 total (0 active), Execution time: mean = 22.042 us, total = 34.297 ms, Queueing time: mean = 12.426 us, max = 137.899 us, min = 2.227 us, total = 19.334 ms InternalKVGcsService.grpc_server.InternalKVGet - 1555 total (0 active), Execution time: mean = 567.341 us, total = 882.215 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1555 total (0 active), Execution time: mean = 55.408 us, total = 86.160 ms, Queueing time: mean = 136.072 us, max = 22.982 ms, min = 9.184 us, total = 211.592 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1496 total (0 active), Execution time: mean = 615.161 us, total = 920.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1496 total (0 active), Execution time: mean = 90.651 us, total = 135.613 ms, Queueing time: mean = 105.041 us, max = 273.843 us, min = 15.511 us, total = 157.141 ms GCSServer.deadline_timer.debug_state_dump - 750 total (1 active), Execution time: mean = 1.352 ms, total = 1.014 s, Queueing time: mean = 36.830 us, max = 178.374 us, min = -0.023 s, total = 27.623 ms GCSServer.deadline_timer.debug_state_event_stats_print - 125 total (1 active, 1 running), Execution time: mean = 1.520 ms, total = 190.050 ms, Queueing time: mean = 58.417 us, max = 238.837 us, min = 13.943 us, total = 7.302 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:17:15,386 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 346358 total (1 active) Queueing time: mean = 39.618 us, max = 58.960 ms, min = 1.891 us, total = 13.722 s Execution time: mean = 212.325 us, total = 73.540 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 157433 total (0 active), Execution time: mean = 439.571 us, total = 69.203 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 157433 total (0 active), Execution time: mean = 22.546 us, total = 3.549 s, Queueing time: mean = 84.754 us, max = 58.960 ms, min = 2.005 us, total = 13.343 s event_loop_lag_probe - 29990 total (0 active), Execution time: mean = 25.799 us, total = 773.703 ms, Queueing time: mean = 8.978 us, max = 395.058 us, min = 1.891 us, total = 269.263 ms GcsTaskManager.GcJobSummary - 1501 total (1 active), Execution time: mean = 9.505 us, total = 14.267 ms, Queueing time: mean = 73.075 us, max = 3.642 ms, min = 12.167 us, total = 109.685 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:17:15,386 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36914 total (25 active) Queueing time: mean = 15.423 us, max = 600.706 us, min = 1.580 us, total = 569.311 ms Execution time: mean = 4.728 s, total = 174546.562 s Event stats: event_loop_lag_probe - 29991 total (0 active), Execution time: mean = 25.519 us, total = 765.347 ms, Queueing time: mean = 9.211 us, max = 600.706 us, min = 1.580 us, total = 276.244 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1864 total (0 active), Execution time: mean = 36.111 us, total = 67.311 ms, Queueing time: mean = 80.984 us, max = 408.208 us, min = 4.084 us, total = 150.954 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1864 total (24 active), Execution time: mean = 93.640 s, total = 174544.678 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1559 total (0 active), Execution time: mean = 101.694 us, total = 158.540 ms, Queueing time: mean = 88.743 us, max = 260.819 us, min = 11.233 us, total = 138.351 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1559 total (0 active), Execution time: mean = 542.271 us, total = 845.400 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 447.304 us, total = 11.630 ms, Queueing time: mean = 68.110 us, max = 114.456 us, min = 33.142 us, total = 1.771 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:17:15,386 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30037 total (0 active) Queueing time: mean = 9.860 us, max = 2.505 ms, min = 563.000 ns, total = 296.161 ms Execution time: mean = 25.947 us, total = 779.361 ms Event stats: event_loop_lag_probe - 29991 total (0 active), Execution time: mean = 25.864 us, total = 775.677 ms, Queueing time: mean = 9.790 us, max = 2.505 ms, min = 1.873 us, total = 293.605 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:18:15,387 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1508 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1572 - cumulative published bytes: 1247064 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:18:15,388 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 90931 total (4 active) Queueing time: mean = 62.207 us, max = 225.583 ms, min = -0.023 s, total = 5.657 s Execution time: mean = 224.193 us, total = 20.386 s Event stats: event_loop_lag_probe - 30229 total (0 active), Execution time: mean = 25.856 us, total = 781.600 ms, Queueing time: mean = 9.955 us, max = 6.895 ms, min = 1.268 us, total = 300.941 ms GcsInMemoryStore.Put - 12135 total (0 active), Execution time: mean = 23.763 us, total = 288.364 ms, Queueing time: mean = 173.737 us, max = 225.278 ms, min = 1.495 us, total = 2.108 s RayletLoadPulled - 7558 total (1 active), Execution time: mean = 242.936 us, total = 1.836 s, Queueing time: mean = 73.867 us, max = 1.351 ms, min = 10.621 us, total = 558.290 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7556 total (0 active), Execution time: mean = 74.815 us, total = 565.301 ms, Queueing time: mean = 98.446 us, max = 15.090 ms, min = 11.600 us, total = 743.857 ms NodeManagerService.grpc_client.GetResourceLoad - 7556 total (0 active), Execution time: mean = 1.350 ms, total = 10.198 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3031 total (0 active), Execution time: mean = 486.780 us, total = 1.475 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3031 total (0 active), Execution time: mean = 39.937 us, total = 121.049 ms, Queueing time: mean = 89.778 us, max = 601.063 us, min = 9.986 us, total = 272.118 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3023 total (0 active), Execution time: mean = 34.805 us, total = 105.215 ms, Queueing time: mean = 105.628 us, max = 1.496 ms, min = 14.438 us, total = 319.313 ms NodeInfoGcsService.grpc_server.CheckAlive - 3023 total (0 active), Execution time: mean = 509.689 us, total = 1.541 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2520 total (1 active), Execution time: mean = 10.260 us, total = 25.855 ms, Queueing time: mean = 74.226 us, max = 1.648 ms, min = 7.259 us, total = 187.049 ms HealthCheck - 2518 total (0 active), Execution time: mean = 7.556 us, total = 19.026 ms, Queueing time: mean = 90.553 us, max = 1.139 ms, min = 11.335 us, total = 228.014 ms GcsInMemoryStore.Get - 1568 total (0 active), Execution time: mean = 22.038 us, total = 34.555 ms, Queueing time: mean = 12.418 us, max = 137.899 us, min = 2.227 us, total = 19.472 ms InternalKVGcsService.grpc_server.InternalKVGet - 1567 total (0 active), Execution time: mean = 566.746 us, total = 888.091 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1567 total (0 active), Execution time: mean = 55.401 us, total = 86.814 ms, Queueing time: mean = 135.631 us, max = 22.982 ms, min = 9.184 us, total = 212.534 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1508 total (0 active), Execution time: mean = 614.948 us, total = 927.342 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1508 total (0 active), Execution time: mean = 90.714 us, total = 136.796 ms, Queueing time: mean = 105.040 us, max = 273.843 us, min = 15.511 us, total = 158.401 ms GCSServer.deadline_timer.debug_state_dump - 756 total (1 active), Execution time: mean = 1.353 ms, total = 1.023 s, Queueing time: mean = 37.181 us, max = 178.374 us, min = -0.023 s, total = 28.109 ms GCSServer.deadline_timer.debug_state_event_stats_print - 126 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 191.590 ms, Queueing time: mean = 58.391 us, max = 238.837 us, min = 13.943 us, total = 7.357 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:18:15,388 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 349130 total (1 active) Queueing time: mean = 39.595 us, max = 58.960 ms, min = 1.891 us, total = 13.824 s Execution time: mean = 212.276 us, total = 74.112 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 158693 total (0 active), Execution time: mean = 439.466 us, total = 69.740 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 158693 total (0 active), Execution time: mean = 22.545 us, total = 3.578 s, Queueing time: mean = 84.705 us, max = 58.960 ms, min = 2.005 us, total = 13.442 s event_loop_lag_probe - 30230 total (0 active), Execution time: mean = 25.785 us, total = 779.488 ms, Queueing time: mean = 8.974 us, max = 395.058 us, min = 1.891 us, total = 271.295 ms GcsTaskManager.GcJobSummary - 1513 total (1 active), Execution time: mean = 9.506 us, total = 14.383 ms, Queueing time: mean = 73.042 us, max = 3.642 ms, min = 12.167 us, total = 110.513 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:18:15,388 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37206 total (25 active) Queueing time: mean = 15.421 us, max = 600.706 us, min = 1.580 us, total = 573.764 ms Execution time: mean = 4.693 s, total = 174606.715 s Event stats: event_loop_lag_probe - 30231 total (0 active), Execution time: mean = 25.512 us, total = 771.262 ms, Queueing time: mean = 9.208 us, max = 600.706 us, min = 1.580 us, total = 278.357 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1877 total (0 active), Execution time: mean = 36.123 us, total = 67.803 ms, Queueing time: mean = 81.074 us, max = 408.208 us, min = 4.084 us, total = 152.176 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1877 total (24 active), Execution time: mean = 93.023 s, total = 174604.815 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1572 total (0 active), Execution time: mean = 101.782 us, total = 160.002 ms, Queueing time: mean = 88.721 us, max = 260.819 us, min = 11.233 us, total = 139.469 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1572 total (0 active), Execution time: mean = 542.484 us, total = 852.784 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 447.304 us, total = 11.630 ms, Queueing time: mean = 68.110 us, max = 114.456 us, min = 33.142 us, total = 1.771 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:18:15,388 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30277 total (0 active) Queueing time: mean = 9.848 us, max = 2.505 ms, min = 563.000 ns, total = 298.177 ms Execution time: mean = 25.917 us, total = 784.698 ms Event stats: event_loop_lag_probe - 30231 total (0 active), Execution time: mean = 25.835 us, total = 781.013 ms, Queueing time: mean = 9.779 us, max = 2.505 ms, min = 1.873 us, total = 295.621 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:19:15,389 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1520 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1584 - cumulative published bytes: 1256892 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:19:15,389 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 91650 total (4 active) Queueing time: mean = 62.041 us, max = 225.583 ms, min = -0.023 s, total = 5.686 s Execution time: mean = 224.410 us, total = 20.567 s Event stats: event_loop_lag_probe - 30469 total (0 active), Execution time: mean = 25.854 us, total = 787.751 ms, Queueing time: mean = 9.955 us, max = 6.895 ms, min = 1.268 us, total = 303.318 ms GcsInMemoryStore.Put - 12231 total (0 active), Execution time: mean = 23.624 us, total = 288.942 ms, Queueing time: mean = 172.732 us, max = 225.278 ms, min = 1.495 us, total = 2.113 s RayletLoadPulled - 7618 total (1 active), Execution time: mean = 243.142 us, total = 1.852 s, Queueing time: mean = 73.886 us, max = 1.351 ms, min = 10.621 us, total = 562.861 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7616 total (0 active), Execution time: mean = 74.832 us, total = 569.920 ms, Queueing time: mean = 98.540 us, max = 15.090 ms, min = 11.600 us, total = 750.479 ms NodeManagerService.grpc_client.GetResourceLoad - 7616 total (0 active), Execution time: mean = 1.352 ms, total = 10.295 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3055 total (0 active), Execution time: mean = 487.352 us, total = 1.489 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3055 total (0 active), Execution time: mean = 39.929 us, total = 121.984 ms, Queueing time: mean = 89.760 us, max = 601.063 us, min = 9.986 us, total = 274.216 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3047 total (0 active), Execution time: mean = 34.822 us, total = 106.102 ms, Queueing time: mean = 105.710 us, max = 1.496 ms, min = 14.438 us, total = 322.097 ms NodeInfoGcsService.grpc_server.CheckAlive - 3047 total (0 active), Execution time: mean = 510.251 us, total = 1.555 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2540 total (1 active), Execution time: mean = 10.255 us, total = 26.049 ms, Queueing time: mean = 74.470 us, max = 1.648 ms, min = 7.259 us, total = 189.154 ms HealthCheck - 2538 total (0 active), Execution time: mean = 7.560 us, total = 19.188 ms, Queueing time: mean = 90.536 us, max = 1.139 ms, min = 11.335 us, total = 229.779 ms GcsInMemoryStore.Get - 1580 total (0 active), Execution time: mean = 22.056 us, total = 34.849 ms, Queueing time: mean = 12.410 us, max = 137.899 us, min = 2.227 us, total = 19.607 ms InternalKVGcsService.grpc_server.InternalKVGet - 1579 total (0 active), Execution time: mean = 566.785 us, total = 894.954 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1579 total (0 active), Execution time: mean = 55.409 us, total = 87.491 ms, Queueing time: mean = 135.272 us, max = 22.982 ms, min = 9.184 us, total = 213.595 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1520 total (0 active), Execution time: mean = 615.186 us, total = 935.082 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1520 total (0 active), Execution time: mean = 90.753 us, total = 137.945 ms, Queueing time: mean = 104.884 us, max = 273.843 us, min = 15.511 us, total = 159.423 ms GCSServer.deadline_timer.debug_state_dump - 762 total (1 active), Execution time: mean = 1.354 ms, total = 1.032 s, Queueing time: mean = 37.562 us, max = 178.374 us, min = -0.023 s, total = 28.623 ms GCSServer.deadline_timer.debug_state_event_stats_print - 127 total (1 active, 1 running), Execution time: mean = 1.523 ms, total = 193.468 ms, Queueing time: mean = 58.425 us, max = 238.837 us, min = 13.943 us, total = 7.420 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:19:15,390 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 351902 total (1 active) Queueing time: mean = 39.682 us, max = 58.960 ms, min = 1.891 us, total = 13.964 s Execution time: mean = 212.668 us, total = 74.838 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 159953 total (0 active), Execution time: mean = 440.284 us, total = 70.425 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 159953 total (0 active), Execution time: mean = 22.586 us, total = 3.613 s, Queueing time: mean = 84.894 us, max = 58.960 ms, min = 2.005 us, total = 13.579 s event_loop_lag_probe - 30470 total (0 active), Execution time: mean = 25.806 us, total = 786.305 ms, Queueing time: mean = 8.980 us, max = 395.058 us, min = 1.891 us, total = 273.631 ms GcsTaskManager.GcJobSummary - 1525 total (1 active), Execution time: mean = 9.519 us, total = 14.517 ms, Queueing time: mean = 73.095 us, max = 3.642 ms, min = 12.167 us, total = 111.471 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:19:15,390 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37493 total (25 active) Queueing time: mean = 15.433 us, max = 600.706 us, min = 1.580 us, total = 578.630 ms Execution time: mean = 4.659 s, total = 174666.958 s Event stats: event_loop_lag_probe - 30470 total (0 active), Execution time: mean = 25.529 us, total = 777.860 ms, Queueing time: mean = 9.213 us, max = 600.706 us, min = 1.580 us, total = 280.719 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1889 total (0 active), Execution time: mean = 36.191 us, total = 68.365 ms, Queueing time: mean = 81.238 us, max = 408.208 us, min = 4.084 us, total = 153.459 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1889 total (24 active), Execution time: mean = 92.464 s, total = 174665.042 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1584 total (0 active), Execution time: mean = 101.832 us, total = 161.302 ms, Queueing time: mean = 88.819 us, max = 260.819 us, min = 11.233 us, total = 140.690 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1584 total (0 active), Execution time: mean = 543.245 us, total = 860.500 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 447.304 us, total = 11.630 ms, Queueing time: mean = 68.110 us, max = 114.456 us, min = 33.142 us, total = 1.771 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:19:15,390 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30517 total (0 active) Queueing time: mean = 9.842 us, max = 2.505 ms, min = 563.000 ns, total = 300.335 ms Execution time: mean = 25.901 us, total = 790.420 ms Event stats: event_loop_lag_probe - 30471 total (0 active), Execution time: mean = 25.819 us, total = 786.736 ms, Queueing time: mean = 9.773 us, max = 2.505 ms, min = 1.873 us, total = 297.779 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:20:15,390 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1532 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1596 - cumulative published bytes: 1266720 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:20:15,391 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 92369 total (4 active) Queueing time: mean = 61.885 us, max = 225.583 ms, min = -0.023 s, total = 5.716 s Execution time: mean = 224.583 us, total = 20.744 s Event stats: event_loop_lag_probe - 30709 total (0 active), Execution time: mean = 25.849 us, total = 793.797 ms, Queueing time: mean = 9.955 us, max = 6.895 ms, min = 1.268 us, total = 305.695 ms GcsInMemoryStore.Put - 12327 total (0 active), Execution time: mean = 23.491 us, total = 289.568 ms, Queueing time: mean = 171.705 us, max = 225.278 ms, min = 1.495 us, total = 2.117 s RayletLoadPulled - 7678 total (1 active), Execution time: mean = 243.312 us, total = 1.868 s, Queueing time: mean = 73.898 us, max = 1.351 ms, min = 10.621 us, total = 567.392 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7676 total (0 active), Execution time: mean = 74.881 us, total = 574.790 ms, Queueing time: mean = 98.633 us, max = 15.090 ms, min = 11.600 us, total = 757.108 ms NodeManagerService.grpc_client.GetResourceLoad - 7676 total (0 active), Execution time: mean = 1.353 ms, total = 10.389 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3079 total (0 active), Execution time: mean = 487.928 us, total = 1.502 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3079 total (0 active), Execution time: mean = 39.919 us, total = 122.910 ms, Queueing time: mean = 89.933 us, max = 601.063 us, min = 9.986 us, total = 276.905 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3071 total (0 active), Execution time: mean = 34.835 us, total = 106.978 ms, Queueing time: mean = 105.815 us, max = 1.496 ms, min = 14.438 us, total = 324.958 ms NodeInfoGcsService.grpc_server.CheckAlive - 3071 total (0 active), Execution time: mean = 510.678 us, total = 1.568 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2560 total (1 active), Execution time: mean = 10.253 us, total = 26.248 ms, Queueing time: mean = 74.488 us, max = 1.648 ms, min = 7.259 us, total = 190.690 ms HealthCheck - 2558 total (0 active), Execution time: mean = 7.577 us, total = 19.383 ms, Queueing time: mean = 90.658 us, max = 1.139 ms, min = 11.335 us, total = 231.904 ms GcsInMemoryStore.Get - 1592 total (0 active), Execution time: mean = 22.090 us, total = 35.167 ms, Queueing time: mean = 12.415 us, max = 137.899 us, min = 2.227 us, total = 19.765 ms InternalKVGcsService.grpc_server.InternalKVGet - 1591 total (0 active), Execution time: mean = 566.823 us, total = 901.815 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1591 total (0 active), Execution time: mean = 55.397 us, total = 88.136 ms, Queueing time: mean = 135.162 us, max = 22.982 ms, min = 9.184 us, total = 215.043 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1532 total (0 active), Execution time: mean = 615.449 us, total = 942.868 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1532 total (0 active), Execution time: mean = 90.754 us, total = 139.035 ms, Queueing time: mean = 105.027 us, max = 273.843 us, min = 15.511 us, total = 160.901 ms GCSServer.deadline_timer.debug_state_dump - 768 total (1 active), Execution time: mean = 1.355 ms, total = 1.041 s, Queueing time: mean = 37.795 us, max = 178.374 us, min = -0.023 s, total = 29.027 ms GCSServer.deadline_timer.debug_state_event_stats_print - 128 total (1 active, 1 running), Execution time: mean = 1.523 ms, total = 194.946 ms, Queueing time: mean = 58.515 us, max = 238.837 us, min = 13.943 us, total = 7.490 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:20:15,391 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 354662 total (1 active) Queueing time: mean = 39.766 us, max = 58.960 ms, min = 1.891 us, total = 14.104 s Execution time: mean = 213.021 us, total = 75.550 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 161207 total (0 active), Execution time: mean = 441.028 us, total = 71.097 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 161207 total (0 active), Execution time: mean = 22.616 us, total = 3.646 s, Queueing time: mean = 85.078 us, max = 58.960 ms, min = 2.005 us, total = 13.715 s event_loop_lag_probe - 30710 total (0 active), Execution time: mean = 25.820 us, total = 792.932 ms, Queueing time: mean = 8.987 us, max = 395.058 us, min = 1.891 us, total = 275.988 ms GcsTaskManager.GcJobSummary - 1537 total (1 active), Execution time: mean = 9.516 us, total = 14.625 ms, Queueing time: mean = 73.053 us, max = 3.642 ms, min = 12.167 us, total = 112.282 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:20:15,391 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37781 total (25 active) Queueing time: mean = 15.455 us, max = 600.706 us, min = 1.580 us, total = 583.891 ms Execution time: mean = 4.625 s, total = 174727.104 s Event stats: event_loop_lag_probe - 30710 total (0 active), Execution time: mean = 25.544 us, total = 784.442 ms, Queueing time: mean = 9.218 us, max = 600.706 us, min = 1.580 us, total = 283.084 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1901 total (0 active), Execution time: mean = 36.226 us, total = 68.865 ms, Queueing time: mean = 81.478 us, max = 408.208 us, min = 4.084 us, total = 154.889 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1901 total (24 active), Execution time: mean = 91.912 s, total = 174725.172 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1596 total (0 active), Execution time: mean = 101.846 us, total = 162.546 ms, Queueing time: mean = 89.071 us, max = 260.819 us, min = 11.233 us, total = 142.157 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1596 total (0 active), Execution time: mean = 543.950 us, total = 868.144 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 447.304 us, total = 11.630 ms, Queueing time: mean = 68.110 us, max = 114.456 us, min = 33.142 us, total = 1.771 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:20:15,391 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30757 total (0 active) Queueing time: mean = 9.833 us, max = 2.505 ms, min = 563.000 ns, total = 302.444 ms Execution time: mean = 25.880 us, total = 795.982 ms Event stats: event_loop_lag_probe - 30711 total (0 active), Execution time: mean = 25.798 us, total = 792.298 ms, Queueing time: mean = 9.765 us, max = 2.505 ms, min = 1.873 us, total = 299.888 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:21:15,392 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1544 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1608 - cumulative published bytes: 1276548 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:21:15,393 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 93088 total (4 active) Queueing time: mean = 61.734 us, max = 225.583 ms, min = -0.023 s, total = 5.747 s Execution time: mean = 224.729 us, total = 20.920 s Event stats: event_loop_lag_probe - 30949 total (0 active), Execution time: mean = 25.842 us, total = 799.773 ms, Queueing time: mean = 9.952 us, max = 6.895 ms, min = 1.268 us, total = 307.995 ms GcsInMemoryStore.Put - 12423 total (0 active), Execution time: mean = 23.361 us, total = 290.213 ms, Queueing time: mean = 170.732 us, max = 225.278 ms, min = 1.495 us, total = 2.121 s RayletLoadPulled - 7738 total (1 active), Execution time: mean = 243.415 us, total = 1.884 s, Queueing time: mean = 73.915 us, max = 1.351 ms, min = 10.621 us, total = 571.956 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7736 total (0 active), Execution time: mean = 74.920 us, total = 579.584 ms, Queueing time: mean = 98.739 us, max = 15.090 ms, min = 11.600 us, total = 763.843 ms NodeManagerService.grpc_client.GetResourceLoad - 7736 total (0 active), Execution time: mean = 1.355 ms, total = 10.481 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3103 total (0 active), Execution time: mean = 488.505 us, total = 1.516 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3103 total (0 active), Execution time: mean = 39.912 us, total = 123.847 ms, Queueing time: mean = 90.136 us, max = 601.063 us, min = 9.986 us, total = 279.693 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3095 total (0 active), Execution time: mean = 34.847 us, total = 107.850 ms, Queueing time: mean = 105.830 us, max = 1.496 ms, min = 14.438 us, total = 327.543 ms NodeInfoGcsService.grpc_server.CheckAlive - 3095 total (0 active), Execution time: mean = 511.108 us, total = 1.582 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2580 total (1 active), Execution time: mean = 10.252 us, total = 26.450 ms, Queueing time: mean = 74.471 us, max = 1.648 ms, min = 7.259 us, total = 192.135 ms HealthCheck - 2578 total (0 active), Execution time: mean = 7.585 us, total = 19.555 ms, Queueing time: mean = 90.757 us, max = 1.139 ms, min = 11.335 us, total = 233.973 ms GcsInMemoryStore.Get - 1604 total (0 active), Execution time: mean = 22.113 us, total = 35.470 ms, Queueing time: mean = 12.421 us, max = 137.899 us, min = 2.227 us, total = 19.924 ms InternalKVGcsService.grpc_server.InternalKVGet - 1603 total (0 active), Execution time: mean = 567.036 us, total = 908.959 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1603 total (0 active), Execution time: mean = 55.399 us, total = 88.804 ms, Queueing time: mean = 135.039 us, max = 22.982 ms, min = 9.184 us, total = 216.468 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1544 total (0 active), Execution time: mean = 615.803 us, total = 950.800 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1544 total (0 active), Execution time: mean = 90.823 us, total = 140.230 ms, Queueing time: mean = 105.191 us, max = 273.843 us, min = 15.511 us, total = 162.415 ms GCSServer.deadline_timer.debug_state_dump - 774 total (1 active), Execution time: mean = 1.355 ms, total = 1.049 s, Queueing time: mean = 37.995 us, max = 178.374 us, min = -0.023 s, total = 29.408 ms GCSServer.deadline_timer.debug_state_event_stats_print - 129 total (1 active, 1 running), Execution time: mean = 1.522 ms, total = 196.305 ms, Queueing time: mean = 58.687 us, max = 238.837 us, min = 13.943 us, total = 7.571 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:21:15,393 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 357432 total (1 active) Queueing time: mean = 39.855 us, max = 58.960 ms, min = 1.891 us, total = 14.245 s Execution time: mean = 213.360 us, total = 76.262 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 162466 total (0 active), Execution time: mean = 441.743 us, total = 71.768 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 162466 total (0 active), Execution time: mean = 22.650 us, total = 3.680 s, Queueing time: mean = 85.273 us, max = 58.960 ms, min = 2.005 us, total = 13.854 s event_loop_lag_probe - 30950 total (0 active), Execution time: mean = 25.821 us, total = 799.152 ms, Queueing time: mean = 8.988 us, max = 395.058 us, min = 1.891 us, total = 278.187 ms GcsTaskManager.GcJobSummary - 1549 total (1 active), Execution time: mean = 9.516 us, total = 14.740 ms, Queueing time: mean = 73.110 us, max = 3.642 ms, min = 12.167 us, total = 113.247 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:21:15,393 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38069 total (25 active) Queueing time: mean = 15.466 us, max = 600.706 us, min = 1.580 us, total = 588.789 ms Execution time: mean = 4.591 s, total = 174787.306 s Event stats: event_loop_lag_probe - 30950 total (0 active), Execution time: mean = 25.549 us, total = 790.738 ms, Queueing time: mean = 9.220 us, max = 600.706 us, min = 1.580 us, total = 285.355 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1913 total (0 active), Execution time: mean = 36.258 us, total = 69.362 ms, Queueing time: mean = 81.635 us, max = 408.208 us, min = 4.084 us, total = 156.167 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1913 total (24 active), Execution time: mean = 91.367 s, total = 174785.358 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1608 total (0 active), Execution time: mean = 101.987 us, total = 163.995 ms, Queueing time: mean = 89.244 us, max = 260.819 us, min = 11.233 us, total = 143.505 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1608 total (0 active), Execution time: mean = 544.791 us, total = 876.024 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 447.304 us, total = 11.630 ms, Queueing time: mean = 68.110 us, max = 114.456 us, min = 33.142 us, total = 1.771 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:21:15,393 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30997 total (0 active) Queueing time: mean = 9.825 us, max = 2.505 ms, min = 563.000 ns, total = 304.548 ms Execution time: mean = 25.859 us, total = 801.543 ms Event stats: event_loop_lag_probe - 30951 total (0 active), Execution time: mean = 25.778 us, total = 797.859 ms, Queueing time: mean = 9.757 us, max = 2.505 ms, min = 1.873 us, total = 301.992 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:22:15,393 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1556 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1620 - cumulative published bytes: 1286376 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:22:15,394 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 93807 total (4 active) Queueing time: mean = 61.584 us, max = 225.583 ms, min = -0.023 s, total = 5.777 s Execution time: mean = 224.883 us, total = 21.096 s Event stats: event_loop_lag_probe - 31189 total (0 active), Execution time: mean = 25.824 us, total = 805.432 ms, Queueing time: mean = 9.947 us, max = 6.895 ms, min = 1.268 us, total = 310.232 ms GcsInMemoryStore.Put - 12519 total (0 active), Execution time: mean = 23.237 us, total = 290.901 ms, Queueing time: mean = 169.821 us, max = 225.278 ms, min = 1.495 us, total = 2.126 s RayletLoadPulled - 7798 total (1 active), Execution time: mean = 243.580 us, total = 1.899 s, Queueing time: mean = 73.894 us, max = 1.351 ms, min = 10.621 us, total = 576.224 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7796 total (0 active), Execution time: mean = 74.951 us, total = 584.316 ms, Queueing time: mean = 98.851 us, max = 15.090 ms, min = 11.600 us, total = 770.644 ms NodeManagerService.grpc_client.GetResourceLoad - 7796 total (0 active), Execution time: mean = 1.356 ms, total = 10.573 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3127 total (0 active), Execution time: mean = 489.103 us, total = 1.529 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3127 total (0 active), Execution time: mean = 39.947 us, total = 124.915 ms, Queueing time: mean = 90.238 us, max = 601.063 us, min = 9.986 us, total = 282.175 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3119 total (0 active), Execution time: mean = 34.895 us, total = 108.838 ms, Queueing time: mean = 105.843 us, max = 1.496 ms, min = 14.438 us, total = 330.124 ms NodeInfoGcsService.grpc_server.CheckAlive - 3119 total (0 active), Execution time: mean = 511.525 us, total = 1.595 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2600 total (1 active), Execution time: mean = 10.250 us, total = 26.649 ms, Queueing time: mean = 74.496 us, max = 1.648 ms, min = 7.259 us, total = 193.689 ms HealthCheck - 2598 total (0 active), Execution time: mean = 7.591 us, total = 19.722 ms, Queueing time: mean = 90.859 us, max = 1.139 ms, min = 11.335 us, total = 236.053 ms GcsInMemoryStore.Get - 1616 total (0 active), Execution time: mean = 22.160 us, total = 35.811 ms, Queueing time: mean = 12.423 us, max = 137.899 us, min = 2.227 us, total = 20.075 ms InternalKVGcsService.grpc_server.InternalKVGet - 1615 total (0 active), Execution time: mean = 567.024 us, total = 915.744 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1615 total (0 active), Execution time: mean = 55.355 us, total = 89.398 ms, Queueing time: mean = 134.780 us, max = 22.982 ms, min = 9.184 us, total = 217.670 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1556 total (0 active), Execution time: mean = 616.147 us, total = 958.725 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1556 total (0 active), Execution time: mean = 90.902 us, total = 141.443 ms, Queueing time: mean = 105.312 us, max = 273.843 us, min = 15.511 us, total = 163.866 ms GCSServer.deadline_timer.debug_state_dump - 780 total (1 active), Execution time: mean = 1.356 ms, total = 1.058 s, Queueing time: mean = 38.280 us, max = 178.374 us, min = -0.023 s, total = 29.858 ms GCSServer.deadline_timer.debug_state_event_stats_print - 130 total (1 active, 1 running), Execution time: mean = 1.524 ms, total = 198.075 ms, Queueing time: mean = 58.733 us, max = 238.837 us, min = 13.943 us, total = 7.635 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:22:15,395 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 360204 total (1 active) Queueing time: mean = 39.939 us, max = 58.960 ms, min = 1.891 us, total = 14.386 s Execution time: mean = 213.732 us, total = 76.987 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 163726 total (0 active), Execution time: mean = 442.514 us, total = 72.451 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 163726 total (0 active), Execution time: mean = 22.698 us, total = 3.716 s, Queueing time: mean = 85.459 us, max = 58.960 ms, min = 2.005 us, total = 13.992 s event_loop_lag_probe - 31190 total (0 active), Execution time: mean = 25.810 us, total = 805.011 ms, Queueing time: mean = 8.984 us, max = 395.058 us, min = 1.891 us, total = 280.208 ms GcsTaskManager.GcJobSummary - 1561 total (1 active), Execution time: mean = 9.517 us, total = 14.857 ms, Queueing time: mean = 73.116 us, max = 3.642 ms, min = 12.167 us, total = 114.135 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:22:15,395 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38394 total (25 active) Queueing time: mean = 15.515 us, max = 600.706 us, min = 1.580 us, total = 595.686 ms Execution time: mean = 4.835 s, total = 185647.412 s Event stats: event_loop_lag_probe - 31190 total (0 active), Execution time: mean = 25.548 us, total = 796.837 ms, Queueing time: mean = 9.222 us, max = 600.706 us, min = 1.580 us, total = 287.621 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1943 total (0 active), Execution time: mean = 36.209 us, total = 70.354 ms, Queueing time: mean = 82.057 us, max = 408.208 us, min = 4.084 us, total = 159.436 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1943 total (24 active), Execution time: mean = 95.546 s, total = 185645.448 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1620 total (0 active), Execution time: mean = 102.045 us, total = 165.312 ms, Queueing time: mean = 89.389 us, max = 260.819 us, min = 11.233 us, total = 144.809 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1620 total (0 active), Execution time: mean = 545.364 us, total = 883.490 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 452.434 us, total = 12.216 ms, Queueing time: mean = 67.733 us, max = 114.456 us, min = 33.142 us, total = 1.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:22:15,395 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31237 total (0 active) Queueing time: mean = 9.819 us, max = 2.505 ms, min = 563.000 ns, total = 306.708 ms Execution time: mean = 25.850 us, total = 807.473 ms Event stats: event_loop_lag_probe - 31191 total (0 active), Execution time: mean = 25.770 us, total = 803.788 ms, Queueing time: mean = 9.751 us, max = 2.505 ms, min = 1.873 us, total = 304.152 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:23:15,395 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1568 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1633 - cumulative published bytes: 1296238 - current buffered bytes: 153 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:23:15,396 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 94526 total (4 active) Queueing time: mean = 61.431 us, max = 225.583 ms, min = -0.023 s, total = 5.807 s Execution time: mean = 225.052 us, total = 21.273 s Event stats: event_loop_lag_probe - 31429 total (0 active), Execution time: mean = 25.810 us, total = 811.194 ms, Queueing time: mean = 9.943 us, max = 6.895 ms, min = 1.268 us, total = 312.499 ms GcsInMemoryStore.Put - 12615 total (0 active), Execution time: mean = 23.114 us, total = 291.578 ms, Queueing time: mean = 168.886 us, max = 225.278 ms, min = 1.495 us, total = 2.131 s RayletLoadPulled - 7858 total (1 active), Execution time: mean = 243.760 us, total = 1.915 s, Queueing time: mean = 73.934 us, max = 1.351 ms, min = 10.621 us, total = 580.971 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7856 total (0 active), Execution time: mean = 74.981 us, total = 589.051 ms, Queueing time: mean = 98.947 us, max = 15.090 ms, min = 11.600 us, total = 777.326 ms NodeManagerService.grpc_client.GetResourceLoad - 7856 total (0 active), Execution time: mean = 1.358 ms, total = 10.665 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3151 total (0 active), Execution time: mean = 489.842 us, total = 1.543 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3151 total (0 active), Execution time: mean = 39.983 us, total = 125.986 ms, Queueing time: mean = 90.287 us, max = 601.063 us, min = 9.986 us, total = 284.493 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3143 total (0 active), Execution time: mean = 34.950 us, total = 109.848 ms, Queueing time: mean = 105.928 us, max = 1.496 ms, min = 14.438 us, total = 332.933 ms NodeInfoGcsService.grpc_server.CheckAlive - 3143 total (0 active), Execution time: mean = 511.985 us, total = 1.609 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2620 total (1 active), Execution time: mean = 10.243 us, total = 26.837 ms, Queueing time: mean = 74.534 us, max = 1.648 ms, min = 7.259 us, total = 195.279 ms HealthCheck - 2618 total (0 active), Execution time: mean = 7.599 us, total = 19.893 ms, Queueing time: mean = 90.928 us, max = 1.139 ms, min = 11.335 us, total = 238.048 ms GcsInMemoryStore.Get - 1628 total (0 active), Execution time: mean = 22.185 us, total = 36.117 ms, Queueing time: mean = 12.425 us, max = 137.899 us, min = 2.227 us, total = 20.228 ms InternalKVGcsService.grpc_server.InternalKVGet - 1627 total (0 active), Execution time: mean = 567.161 us, total = 922.771 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1627 total (0 active), Execution time: mean = 55.358 us, total = 90.067 ms, Queueing time: mean = 134.476 us, max = 22.982 ms, min = 9.184 us, total = 218.792 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1568 total (0 active), Execution time: mean = 616.156 us, total = 966.132 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1568 total (0 active), Execution time: mean = 90.951 us, total = 142.611 ms, Queueing time: mean = 105.242 us, max = 273.843 us, min = 15.511 us, total = 165.020 ms GCSServer.deadline_timer.debug_state_dump - 786 total (1 active), Execution time: mean = 1.358 ms, total = 1.067 s, Queueing time: mean = 38.546 us, max = 178.374 us, min = -0.023 s, total = 30.297 ms GCSServer.deadline_timer.debug_state_event_stats_print - 131 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 199.773 ms, Queueing time: mean = 58.839 us, max = 238.837 us, min = 13.943 us, total = 7.708 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:23:15,396 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 362973 total (1 active) Queueing time: mean = 40.022 us, max = 58.960 ms, min = 1.891 us, total = 14.527 s Execution time: mean = 214.096 us, total = 77.711 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 164985 total (0 active), Execution time: mean = 443.279 us, total = 73.134 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 164985 total (0 active), Execution time: mean = 22.734 us, total = 3.751 s, Queueing time: mean = 85.641 us, max = 58.960 ms, min = 2.005 us, total = 14.129 s event_loop_lag_probe - 31429 total (0 active), Execution time: mean = 25.797 us, total = 810.777 ms, Queueing time: mean = 8.980 us, max = 395.058 us, min = 1.891 us, total = 282.237 ms GcsTaskManager.GcJobSummary - 1573 total (1 active), Execution time: mean = 9.521 us, total = 14.976 ms, Queueing time: mean = 73.127 us, max = 3.642 ms, min = 12.167 us, total = 115.029 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:23:15,396 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38686 total (25 active) Queueing time: mean = 15.532 us, max = 600.706 us, min = 1.580 us, total = 600.866 ms Execution time: mean = 4.800 s, total = 185707.575 s Event stats: event_loop_lag_probe - 31430 total (0 active), Execution time: mean = 25.556 us, total = 803.227 ms, Queueing time: mean = 9.223 us, max = 600.706 us, min = 1.580 us, total = 289.876 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1956 total (0 active), Execution time: mean = 36.317 us, total = 71.037 ms, Queueing time: mean = 82.205 us, max = 408.208 us, min = 4.084 us, total = 160.793 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1956 total (24 active), Execution time: mean = 94.942 s, total = 185705.593 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1633 total (0 active), Execution time: mean = 102.272 us, total = 167.010 ms, Queueing time: mean = 89.637 us, max = 260.819 us, min = 11.233 us, total = 146.377 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1633 total (0 active), Execution time: mean = 546.350 us, total = 892.190 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 452.434 us, total = 12.216 ms, Queueing time: mean = 67.733 us, max = 114.456 us, min = 33.142 us, total = 1.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:23:15,397 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31476 total (0 active) Queueing time: mean = 9.816 us, max = 2.505 ms, min = 563.000 ns, total = 308.962 ms Execution time: mean = 25.850 us, total = 813.668 ms Event stats: event_loop_lag_probe - 31430 total (0 active), Execution time: mean = 25.771 us, total = 809.984 ms, Queueing time: mean = 9.749 us, max = 2.505 ms, min = 1.873 us, total = 306.406 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:24:15,397 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1580 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1645 - cumulative published bytes: 1306066 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:24:15,397 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 95245 total (4 active) Queueing time: mean = 61.290 us, max = 225.583 ms, min = -0.023 s, total = 5.838 s Execution time: mean = 225.274 us, total = 21.456 s Event stats: event_loop_lag_probe - 31669 total (0 active), Execution time: mean = 25.822 us, total = 817.772 ms, Queueing time: mean = 9.948 us, max = 6.895 ms, min = 1.268 us, total = 315.054 ms GcsInMemoryStore.Put - 12711 total (0 active), Execution time: mean = 22.990 us, total = 292.220 ms, Queueing time: mean = 167.911 us, max = 225.278 ms, min = 1.495 us, total = 2.134 s RayletLoadPulled - 7918 total (1 active), Execution time: mean = 244.017 us, total = 1.932 s, Queueing time: mean = 73.995 us, max = 1.351 ms, min = 10.621 us, total = 585.893 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7916 total (0 active), Execution time: mean = 75.032 us, total = 593.956 ms, Queueing time: mean = 99.062 us, max = 15.090 ms, min = 11.600 us, total = 784.176 ms NodeManagerService.grpc_client.GetResourceLoad - 7916 total (0 active), Execution time: mean = 1.359 ms, total = 10.762 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3175 total (0 active), Execution time: mean = 490.484 us, total = 1.557 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3175 total (0 active), Execution time: mean = 39.964 us, total = 126.886 ms, Queueing time: mean = 90.378 us, max = 601.063 us, min = 9.986 us, total = 286.950 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3167 total (0 active), Execution time: mean = 34.993 us, total = 110.822 ms, Queueing time: mean = 106.014 us, max = 1.496 ms, min = 14.438 us, total = 335.747 ms NodeInfoGcsService.grpc_server.CheckAlive - 3167 total (0 active), Execution time: mean = 512.466 us, total = 1.623 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2640 total (1 active), Execution time: mean = 10.241 us, total = 27.037 ms, Queueing time: mean = 74.573 us, max = 1.648 ms, min = 7.259 us, total = 196.871 ms HealthCheck - 2638 total (0 active), Execution time: mean = 7.608 us, total = 20.069 ms, Queueing time: mean = 91.116 us, max = 1.139 ms, min = 11.335 us, total = 240.365 ms GcsInMemoryStore.Get - 1640 total (0 active), Execution time: mean = 22.221 us, total = 36.442 ms, Queueing time: mean = 12.430 us, max = 137.899 us, min = 2.227 us, total = 20.385 ms InternalKVGcsService.grpc_server.InternalKVGet - 1639 total (0 active), Execution time: mean = 567.595 us, total = 930.289 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1639 total (0 active), Execution time: mean = 55.339 us, total = 90.700 ms, Queueing time: mean = 134.391 us, max = 22.982 ms, min = 9.184 us, total = 220.267 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1580 total (0 active), Execution time: mean = 616.608 us, total = 974.240 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1580 total (0 active), Execution time: mean = 90.951 us, total = 143.703 ms, Queueing time: mean = 105.282 us, max = 273.843 us, min = 15.511 us, total = 166.345 ms GCSServer.deadline_timer.debug_state_dump - 792 total (1 active), Execution time: mean = 1.359 ms, total = 1.076 s, Queueing time: mean = 38.762 us, max = 178.374 us, min = -0.023 s, total = 30.699 ms GCSServer.deadline_timer.debug_state_event_stats_print - 132 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 201.311 ms, Queueing time: mean = 58.930 us, max = 238.837 us, min = 13.943 us, total = 7.779 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:24:15,397 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 365745 total (1 active) Queueing time: mean = 40.112 us, max = 58.960 ms, min = 1.891 us, total = 14.671 s Execution time: mean = 214.455 us, total = 78.436 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 166245 total (0 active), Execution time: mean = 444.037 us, total = 73.819 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 166245 total (0 active), Execution time: mean = 22.767 us, total = 3.785 s, Queueing time: mean = 85.839 us, max = 58.960 ms, min = 2.005 us, total = 14.270 s event_loop_lag_probe - 31669 total (0 active), Execution time: mean = 25.795 us, total = 816.890 ms, Queueing time: mean = 8.978 us, max = 395.058 us, min = 1.891 us, total = 284.338 ms GcsTaskManager.GcJobSummary - 1585 total (1 active), Execution time: mean = 9.549 us, total = 15.135 ms, Queueing time: mean = 73.152 us, max = 3.642 ms, min = 12.167 us, total = 115.947 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:24:15,397 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38974 total (25 active) Queueing time: mean = 15.546 us, max = 600.706 us, min = 1.580 us, total = 605.903 ms Execution time: mean = 4.766 s, total = 185767.772 s Event stats: event_loop_lag_probe - 31670 total (0 active), Execution time: mean = 25.578 us, total = 810.040 ms, Queueing time: mean = 9.232 us, max = 600.706 us, min = 1.580 us, total = 292.386 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1968 total (0 active), Execution time: mean = 36.386 us, total = 71.607 ms, Queueing time: mean = 82.234 us, max = 408.208 us, min = 4.084 us, total = 161.837 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1968 total (24 active), Execution time: mean = 94.393 s, total = 185765.774 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1645 total (0 active), Execution time: mean = 102.331 us, total = 168.335 ms, Queueing time: mean = 89.885 us, max = 260.819 us, min = 11.233 us, total = 147.860 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1645 total (0 active), Execution time: mean = 547.219 us, total = 900.175 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 452.434 us, total = 12.216 ms, Queueing time: mean = 67.733 us, max = 114.456 us, min = 33.142 us, total = 1.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:24:15,397 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31716 total (0 active) Queueing time: mean = 9.819 us, max = 2.505 ms, min = 563.000 ns, total = 311.430 ms Execution time: mean = 25.861 us, total = 820.222 ms Event stats: event_loop_lag_probe - 31670 total (0 active), Execution time: mean = 25.783 us, total = 816.538 ms, Queueing time: mean = 9.753 us, max = 2.505 ms, min = 1.873 us, total = 308.874 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:25:15,398 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1592 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1658 - cumulative published bytes: 1315928 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:25:15,399 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 95964 total (4 active) Queueing time: mean = 61.141 us, max = 225.583 ms, min = -0.023 s, total = 5.867 s Execution time: mean = 225.412 us, total = 21.631 s Event stats: event_loop_lag_probe - 31909 total (0 active), Execution time: mean = 25.823 us, total = 823.997 ms, Queueing time: mean = 9.948 us, max = 6.895 ms, min = 1.268 us, total = 317.435 ms GcsInMemoryStore.Put - 12807 total (0 active), Execution time: mean = 22.864 us, total = 292.819 ms, Queueing time: mean = 166.999 us, max = 225.278 ms, min = 1.495 us, total = 2.139 s RayletLoadPulled - 7978 total (1 active), Execution time: mean = 244.149 us, total = 1.948 s, Queueing time: mean = 74.045 us, max = 1.351 ms, min = 10.621 us, total = 590.731 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7976 total (0 active), Execution time: mean = 75.031 us, total = 598.451 ms, Queueing time: mean = 99.154 us, max = 15.090 ms, min = 11.600 us, total = 790.854 ms NodeManagerService.grpc_client.GetResourceLoad - 7976 total (0 active), Execution time: mean = 1.361 ms, total = 10.854 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3199 total (0 active), Execution time: mean = 490.953 us, total = 1.571 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3199 total (0 active), Execution time: mean = 39.951 us, total = 127.804 ms, Queueing time: mean = 90.457 us, max = 601.063 us, min = 9.986 us, total = 289.372 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3191 total (0 active), Execution time: mean = 35.003 us, total = 111.694 ms, Queueing time: mean = 106.019 us, max = 1.496 ms, min = 14.438 us, total = 338.308 ms NodeInfoGcsService.grpc_server.CheckAlive - 3191 total (0 active), Execution time: mean = 513.064 us, total = 1.637 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2660 total (1 active), Execution time: mean = 10.231 us, total = 27.215 ms, Queueing time: mean = 74.508 us, max = 1.648 ms, min = 7.259 us, total = 198.192 ms HealthCheck - 2658 total (0 active), Execution time: mean = 7.616 us, total = 20.244 ms, Queueing time: mean = 91.151 us, max = 1.139 ms, min = 11.335 us, total = 242.280 ms GcsInMemoryStore.Get - 1652 total (0 active), Execution time: mean = 22.241 us, total = 36.742 ms, Queueing time: mean = 12.438 us, max = 137.899 us, min = 2.227 us, total = 20.548 ms InternalKVGcsService.grpc_server.InternalKVGet - 1651 total (0 active), Execution time: mean = 567.865 us, total = 937.545 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1651 total (0 active), Execution time: mean = 55.316 us, total = 91.326 ms, Queueing time: mean = 134.108 us, max = 22.982 ms, min = 9.184 us, total = 221.413 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1592 total (0 active), Execution time: mean = 616.926 us, total = 982.146 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1592 total (0 active), Execution time: mean = 90.978 us, total = 144.837 ms, Queueing time: mean = 105.360 us, max = 273.843 us, min = 15.511 us, total = 167.733 ms GCSServer.deadline_timer.debug_state_dump - 798 total (1 active), Execution time: mean = 1.358 ms, total = 1.084 s, Queueing time: mean = 39.014 us, max = 178.374 us, min = -0.023 s, total = 31.133 ms GCSServer.deadline_timer.debug_state_event_stats_print - 133 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 202.010 ms, Queueing time: mean = 58.796 us, max = 238.837 us, min = 13.943 us, total = 7.820 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:25:15,399 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 368517 total (1 active) Queueing time: mean = 40.191 us, max = 58.960 ms, min = 1.891 us, total = 14.811 s Execution time: mean = 214.766 us, total = 79.145 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 167505 total (0 active), Execution time: mean = 444.689 us, total = 74.488 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 167505 total (0 active), Execution time: mean = 22.802 us, total = 3.820 s, Queueing time: mean = 86.015 us, max = 58.960 ms, min = 2.005 us, total = 14.408 s event_loop_lag_probe - 31909 total (0 active), Execution time: mean = 25.776 us, total = 822.501 ms, Queueing time: mean = 8.972 us, max = 395.058 us, min = 1.891 us, total = 286.290 ms GcsTaskManager.GcJobSummary - 1597 total (1 active), Execution time: mean = 9.549 us, total = 15.250 ms, Queueing time: mean = 73.182 us, max = 3.642 ms, min = 12.167 us, total = 116.872 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:25:15,399 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39266 total (25 active) Queueing time: mean = 15.612 us, max = 1.217 ms, min = 1.580 us, total = 613.038 ms Execution time: mean = 4.733 s, total = 185827.999 s Event stats: event_loop_lag_probe - 31910 total (0 active), Execution time: mean = 25.598 us, total = 816.830 ms, Queueing time: mean = 9.270 us, max = 1.017 ms, min = 1.580 us, total = 295.809 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1981 total (0 active), Execution time: mean = 36.435 us, total = 72.178 ms, Queueing time: mean = 82.915 us, max = 1.217 ms, min = 4.084 us, total = 164.254 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1981 total (24 active), Execution time: mean = 93.804 s, total = 185825.983 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1658 total (0 active), Execution time: mean = 102.531 us, total = 169.996 ms, Queueing time: mean = 89.961 us, max = 260.819 us, min = 11.233 us, total = 149.156 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1658 total (0 active), Execution time: mean = 548.168 us, total = 908.862 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 452.434 us, total = 12.216 ms, Queueing time: mean = 67.733 us, max = 114.456 us, min = 33.142 us, total = 1.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:25:15,399 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31956 total (0 active) Queueing time: mean = 9.818 us, max = 2.505 ms, min = 563.000 ns, total = 313.743 ms Execution time: mean = 25.858 us, total = 826.328 ms Event stats: event_loop_lag_probe - 31910 total (0 active), Execution time: mean = 25.780 us, total = 822.644 ms, Queueing time: mean = 9.752 us, max = 2.505 ms, min = 1.873 us, total = 311.187 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:26:15,399 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1604 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1670 - cumulative published bytes: 1325756 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:26:15,400 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 96682 total (4 active) Queueing time: mean = 61.034 us, max = 225.583 ms, min = -0.023 s, total = 5.901 s Execution time: mean = 225.626 us, total = 21.814 s Event stats: event_loop_lag_probe - 32148 total (0 active), Execution time: mean = 25.804 us, total = 829.537 ms, Queueing time: mean = 9.941 us, max = 6.895 ms, min = 1.268 us, total = 319.572 ms GcsInMemoryStore.Put - 12903 total (0 active), Execution time: mean = 22.744 us, total = 293.462 ms, Queueing time: mean = 166.080 us, max = 225.278 ms, min = 1.495 us, total = 2.143 s RayletLoadPulled - 8038 total (1 active), Execution time: mean = 244.282 us, total = 1.964 s, Queueing time: mean = 74.054 us, max = 1.351 ms, min = 10.621 us, total = 595.243 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8036 total (0 active), Execution time: mean = 75.083 us, total = 603.363 ms, Queueing time: mean = 99.766 us, max = 15.090 ms, min = 11.600 us, total = 801.719 ms NodeManagerService.grpc_client.GetResourceLoad - 8036 total (0 active), Execution time: mean = 1.363 ms, total = 10.953 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3223 total (0 active), Execution time: mean = 491.574 us, total = 1.584 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3223 total (0 active), Execution time: mean = 40.006 us, total = 128.939 ms, Queueing time: mean = 90.575 us, max = 601.063 us, min = 9.986 us, total = 291.923 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3215 total (0 active), Execution time: mean = 35.023 us, total = 112.599 ms, Queueing time: mean = 106.068 us, max = 1.496 ms, min = 14.438 us, total = 341.008 ms NodeInfoGcsService.grpc_server.CheckAlive - 3215 total (0 active), Execution time: mean = 513.524 us, total = 1.651 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2680 total (1 active), Execution time: mean = 10.224 us, total = 27.401 ms, Queueing time: mean = 74.471 us, max = 1.648 ms, min = 7.259 us, total = 199.583 ms HealthCheck - 2678 total (0 active), Execution time: mean = 7.621 us, total = 20.409 ms, Queueing time: mean = 91.222 us, max = 1.139 ms, min = 11.335 us, total = 244.294 ms GcsInMemoryStore.Get - 1664 total (0 active), Execution time: mean = 22.264 us, total = 37.048 ms, Queueing time: mean = 12.445 us, max = 137.899 us, min = 2.227 us, total = 20.709 ms InternalKVGcsService.grpc_server.InternalKVGet - 1663 total (0 active), Execution time: mean = 567.924 us, total = 944.458 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1663 total (0 active), Execution time: mean = 55.299 us, total = 91.961 ms, Queueing time: mean = 133.902 us, max = 22.982 ms, min = 9.184 us, total = 222.679 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1604 total (0 active), Execution time: mean = 617.315 us, total = 990.173 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1604 total (0 active), Execution time: mean = 91.025 us, total = 146.003 ms, Queueing time: mean = 105.335 us, max = 273.843 us, min = 15.511 us, total = 168.957 ms GCSServer.deadline_timer.debug_state_dump - 804 total (1 active), Execution time: mean = 1.359 ms, total = 1.093 s, Queueing time: mean = 39.282 us, max = 178.374 us, min = -0.023 s, total = 31.583 ms GCSServer.deadline_timer.debug_state_event_stats_print - 134 total (1 active, 1 running), Execution time: mean = 1.520 ms, total = 203.691 ms, Queueing time: mean = 58.973 us, max = 238.837 us, min = 13.943 us, total = 7.902 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:26:15,401 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 371271 total (1 active) Queueing time: mean = 40.267 us, max = 58.960 ms, min = 1.891 us, total = 14.950 s Execution time: mean = 215.128 us, total = 79.871 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 168756 total (0 active), Execution time: mean = 445.443 us, total = 75.171 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 168756 total (0 active), Execution time: mean = 22.849 us, total = 3.856 s, Queueing time: mean = 86.182 us, max = 58.960 ms, min = 2.005 us, total = 14.544 s event_loop_lag_probe - 32149 total (0 active), Execution time: mean = 25.769 us, total = 828.455 ms, Queueing time: mean = 8.969 us, max = 395.058 us, min = 1.891 us, total = 288.353 ms GcsTaskManager.GcJobSummary - 1609 total (1 active), Execution time: mean = 9.543 us, total = 15.355 ms, Queueing time: mean = 73.196 us, max = 3.642 ms, min = 12.167 us, total = 117.772 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:26:15,401 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39554 total (25 active) Queueing time: mean = 15.618 us, max = 1.217 ms, min = 1.580 us, total = 617.753 ms Execution time: mean = 4.700 s, total = 185888.112 s Event stats: event_loop_lag_probe - 32150 total (0 active), Execution time: mean = 25.604 us, total = 823.180 ms, Queueing time: mean = 9.273 us, max = 1.017 ms, min = 1.580 us, total = 298.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1993 total (0 active), Execution time: mean = 36.461 us, total = 72.666 ms, Queueing time: mean = 83.094 us, max = 1.217 ms, min = 4.084 us, total = 165.606 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1993 total (24 active), Execution time: mean = 93.269 s, total = 185886.081 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1670 total (0 active), Execution time: mean = 102.489 us, total = 171.157 ms, Queueing time: mean = 89.939 us, max = 260.819 us, min = 11.233 us, total = 150.198 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1670 total (0 active), Execution time: mean = 548.661 us, total = 916.263 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 452.434 us, total = 12.216 ms, Queueing time: mean = 67.733 us, max = 114.456 us, min = 33.142 us, total = 1.829 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:26:15,401 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32196 total (0 active) Queueing time: mean = 9.813 us, max = 2.505 ms, min = 563.000 ns, total = 315.935 ms Execution time: mean = 25.844 us, total = 832.083 ms Event stats: event_loop_lag_probe - 32150 total (0 active), Execution time: mean = 25.767 us, total = 828.399 ms, Queueing time: mean = 9.747 us, max = 2.505 ms, min = 1.873 us, total = 313.379 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:27:15,401 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1616 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1682 - cumulative published bytes: 1335584 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:27:15,402 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 97401 total (4 active) Queueing time: mean = 60.885 us, max = 225.583 ms, min = -0.023 s, total = 5.930 s Execution time: mean = 225.758 us, total = 21.989 s Event stats: event_loop_lag_probe - 32388 total (0 active), Execution time: mean = 25.787 us, total = 835.176 ms, Queueing time: mean = 9.934 us, max = 6.895 ms, min = 1.268 us, total = 321.757 ms GcsInMemoryStore.Put - 12999 total (0 active), Execution time: mean = 22.645 us, total = 294.359 ms, Queueing time: mean = 165.173 us, max = 225.278 ms, min = 1.495 us, total = 2.147 s RayletLoadPulled - 8098 total (1 active), Execution time: mean = 244.352 us, total = 1.979 s, Queueing time: mean = 74.013 us, max = 1.351 ms, min = 10.621 us, total = 599.358 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8096 total (0 active), Execution time: mean = 75.104 us, total = 608.038 ms, Queueing time: mean = 99.877 us, max = 15.090 ms, min = 11.600 us, total = 808.608 ms NodeManagerService.grpc_client.GetResourceLoad - 8096 total (0 active), Execution time: mean = 1.364 ms, total = 11.044 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3247 total (0 active), Execution time: mean = 492.179 us, total = 1.598 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3247 total (0 active), Execution time: mean = 40.013 us, total = 129.924 ms, Queueing time: mean = 90.661 us, max = 601.063 us, min = 9.986 us, total = 294.376 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3239 total (0 active), Execution time: mean = 35.066 us, total = 113.579 ms, Queueing time: mean = 106.122 us, max = 1.496 ms, min = 14.438 us, total = 343.729 ms NodeInfoGcsService.grpc_server.CheckAlive - 3239 total (0 active), Execution time: mean = 513.911 us, total = 1.665 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2700 total (1 active), Execution time: mean = 10.218 us, total = 27.587 ms, Queueing time: mean = 74.548 us, max = 1.648 ms, min = 7.259 us, total = 201.279 ms HealthCheck - 2698 total (0 active), Execution time: mean = 7.625 us, total = 20.572 ms, Queueing time: mean = 91.312 us, max = 1.139 ms, min = 11.335 us, total = 246.359 ms GcsInMemoryStore.Get - 1676 total (0 active), Execution time: mean = 22.282 us, total = 37.345 ms, Queueing time: mean = 12.456 us, max = 137.899 us, min = 2.227 us, total = 20.876 ms InternalKVGcsService.grpc_server.InternalKVGet - 1675 total (0 active), Execution time: mean = 568.135 us, total = 951.626 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1675 total (0 active), Execution time: mean = 55.277 us, total = 92.589 ms, Queueing time: mean = 133.699 us, max = 22.982 ms, min = 9.184 us, total = 223.946 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1616 total (0 active), Execution time: mean = 617.554 us, total = 997.967 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1616 total (0 active), Execution time: mean = 91.082 us, total = 147.189 ms, Queueing time: mean = 105.349 us, max = 273.843 us, min = 15.511 us, total = 170.244 ms GCSServer.deadline_timer.debug_state_dump - 810 total (1 active), Execution time: mean = 1.360 ms, total = 1.102 s, Queueing time: mean = 39.483 us, max = 178.374 us, min = -0.023 s, total = 31.981 ms GCSServer.deadline_timer.debug_state_event_stats_print - 135 total (1 active, 1 running), Execution time: mean = 1.522 ms, total = 205.437 ms, Queueing time: mean = 58.873 us, max = 238.837 us, min = 13.943 us, total = 7.948 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:27:15,402 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 374043 total (1 active) Queueing time: mean = 40.351 us, max = 58.960 ms, min = 1.891 us, total = 15.093 s Execution time: mean = 215.480 us, total = 80.599 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 170016 total (0 active), Execution time: mean = 446.170 us, total = 75.856 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 170016 total (0 active), Execution time: mean = 22.898 us, total = 3.893 s, Queueing time: mean = 86.368 us, max = 58.960 ms, min = 2.005 us, total = 14.684 s event_loop_lag_probe - 32389 total (0 active), Execution time: mean = 25.755 us, total = 834.181 ms, Queueing time: mean = 8.968 us, max = 395.058 us, min = 1.891 us, total = 290.452 ms GcsTaskManager.GcJobSummary - 1621 total (1 active), Execution time: mean = 9.544 us, total = 15.470 ms, Queueing time: mean = 73.208 us, max = 3.642 ms, min = 12.167 us, total = 118.671 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:27:15,403 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39853 total (25 active) Queueing time: mean = 15.644 us, max = 1.217 ms, min = 1.580 us, total = 623.475 ms Execution time: mean = 4.741 s, total = 188948.348 s Event stats: event_loop_lag_probe - 32390 total (0 active), Execution time: mean = 25.606 us, total = 829.393 ms, Queueing time: mean = 9.274 us, max = 1.017 ms, min = 1.580 us, total = 300.378 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2010 total (0 active), Execution time: mean = 36.522 us, total = 73.410 ms, Queueing time: mean = 83.407 us, max = 1.217 ms, min = 4.084 us, total = 167.648 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2010 total (24 active), Execution time: mean = 94.003 s, total = 188946.300 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1682 total (0 active), Execution time: mean = 102.507 us, total = 172.416 ms, Queueing time: mean = 90.103 us, max = 260.819 us, min = 11.233 us, total = 151.553 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1682 total (0 active), Execution time: mean = 549.336 us, total = 923.983 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 439.954 us, total = 12.319 ms, Queueing time: mean = 68.003 us, max = 114.456 us, min = 33.142 us, total = 1.904 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:27:15,403 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32436 total (0 active) Queueing time: mean = 9.808 us, max = 2.505 ms, min = 563.000 ns, total = 318.140 ms Execution time: mean = 25.837 us, total = 838.043 ms Event stats: event_loop_lag_probe - 32390 total (0 active), Execution time: mean = 25.760 us, total = 834.359 ms, Queueing time: mean = 9.743 us, max = 2.505 ms, min = 1.873 us, total = 315.584 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:28:15,403 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1628 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1694 - cumulative published bytes: 1345412 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:28:15,403 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 98120 total (4 active) Queueing time: mean = 60.688 us, max = 225.583 ms, min = -0.023 s, total = 5.955 s Execution time: mean = 225.771 us, total = 22.153 s Event stats: event_loop_lag_probe - 32628 total (0 active), Execution time: mean = 25.761 us, total = 840.535 ms, Queueing time: mean = 9.924 us, max = 6.895 ms, min = 1.268 us, total = 323.816 ms GcsInMemoryStore.Put - 13095 total (0 active), Execution time: mean = 22.519 us, total = 294.883 ms, Queueing time: mean = 164.267 us, max = 225.278 ms, min = 1.495 us, total = 2.151 s RayletLoadPulled - 8158 total (1 active), Execution time: mean = 244.361 us, total = 1.993 s, Queueing time: mean = 73.940 us, max = 1.351 ms, min = 10.621 us, total = 603.204 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8156 total (0 active), Execution time: mean = 75.085 us, total = 612.393 ms, Queueing time: mean = 99.805 us, max = 15.090 ms, min = 11.600 us, total = 814.011 ms NodeManagerService.grpc_client.GetResourceLoad - 8156 total (0 active), Execution time: mean = 1.365 ms, total = 11.132 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3271 total (0 active), Execution time: mean = 492.252 us, total = 1.610 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3271 total (0 active), Execution time: mean = 39.968 us, total = 130.735 ms, Queueing time: mean = 90.616 us, max = 601.063 us, min = 9.986 us, total = 296.406 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3263 total (0 active), Execution time: mean = 35.058 us, total = 114.396 ms, Queueing time: mean = 105.987 us, max = 1.496 ms, min = 14.438 us, total = 345.836 ms NodeInfoGcsService.grpc_server.CheckAlive - 3263 total (0 active), Execution time: mean = 513.781 us, total = 1.676 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2720 total (1 active), Execution time: mean = 10.207 us, total = 27.764 ms, Queueing time: mean = 74.493 us, max = 1.648 ms, min = 7.259 us, total = 202.620 ms HealthCheck - 2718 total (0 active), Execution time: mean = 7.618 us, total = 20.705 ms, Queueing time: mean = 91.024 us, max = 1.139 ms, min = 11.335 us, total = 247.404 ms GcsInMemoryStore.Get - 1688 total (0 active), Execution time: mean = 22.274 us, total = 37.599 ms, Queueing time: mean = 12.457 us, max = 137.899 us, min = 2.227 us, total = 21.027 ms InternalKVGcsService.grpc_server.InternalKVGet - 1687 total (0 active), Execution time: mean = 567.689 us, total = 957.691 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1687 total (0 active), Execution time: mean = 55.248 us, total = 93.203 ms, Queueing time: mean = 133.293 us, max = 22.982 ms, min = 9.184 us, total = 224.865 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1628 total (0 active), Execution time: mean = 617.161 us, total = 1.005 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1628 total (0 active), Execution time: mean = 90.996 us, total = 148.142 ms, Queueing time: mean = 105.235 us, max = 273.843 us, min = 15.511 us, total = 171.323 ms GCSServer.deadline_timer.debug_state_dump - 816 total (1 active), Execution time: mean = 1.361 ms, total = 1.110 s, Queueing time: mean = 39.635 us, max = 178.374 us, min = -0.023 s, total = 32.342 ms GCSServer.deadline_timer.debug_state_event_stats_print - 136 total (1 active, 1 running), Execution time: mean = 1.524 ms, total = 207.216 ms, Queueing time: mean = 59.132 us, max = 238.837 us, min = 13.943 us, total = 8.042 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:28:15,404 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 376815 total (1 active) Queueing time: mean = 40.344 us, max = 58.960 ms, min = 1.891 us, total = 15.202 s Execution time: mean = 215.462 us, total = 81.189 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 171276 total (0 active), Execution time: mean = 446.131 us, total = 76.412 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 171276 total (0 active), Execution time: mean = 22.901 us, total = 3.922 s, Queueing time: mean = 86.355 us, max = 58.960 ms, min = 2.005 us, total = 14.791 s event_loop_lag_probe - 32629 total (0 active), Execution time: mean = 25.736 us, total = 839.753 ms, Queueing time: mean = 8.961 us, max = 395.058 us, min = 1.891 us, total = 292.386 ms GcsTaskManager.GcJobSummary - 1633 total (1 active), Execution time: mean = 9.548 us, total = 15.592 ms, Queueing time: mean = 73.103 us, max = 3.642 ms, min = 12.167 us, total = 119.377 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:28:15,404 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40141 total (25 active) Queueing time: mean = 15.645 us, max = 1.217 ms, min = 1.580 us, total = 627.988 ms Execution time: mean = 4.709 s, total = 189008.495 s Event stats: event_loop_lag_probe - 32630 total (0 active), Execution time: mean = 25.605 us, total = 835.495 ms, Queueing time: mean = 9.274 us, max = 1.017 ms, min = 1.580 us, total = 302.598 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2022 total (0 active), Execution time: mean = 36.574 us, total = 73.953 ms, Queueing time: mean = 83.454 us, max = 1.217 ms, min = 4.084 us, total = 168.744 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2022 total (24 active), Execution time: mean = 93.475 s, total = 189006.432 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1694 total (0 active), Execution time: mean = 102.491 us, total = 173.619 ms, Queueing time: mean = 90.172 us, max = 260.819 us, min = 11.233 us, total = 152.751 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1694 total (0 active), Execution time: mean = 549.713 us, total = 931.214 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 439.954 us, total = 12.319 ms, Queueing time: mean = 68.003 us, max = 114.456 us, min = 33.142 us, total = 1.904 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:28:15,404 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32676 total (0 active) Queueing time: mean = 9.802 us, max = 2.505 ms, min = 563.000 ns, total = 320.292 ms Execution time: mean = 25.822 us, total = 843.749 ms Event stats: event_loop_lag_probe - 32630 total (0 active), Execution time: mean = 25.745 us, total = 840.065 ms, Queueing time: mean = 9.738 us, max = 2.505 ms, min = 1.873 us, total = 317.736 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:29:15,404 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1640 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1707 - cumulative published bytes: 1355274 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:29:15,405 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 98839 total (4 active) Queueing time: mean = 60.448 us, max = 225.583 ms, min = -0.023 s, total = 5.975 s Execution time: mean = 225.568 us, total = 22.295 s Event stats: event_loop_lag_probe - 32868 total (0 active), Execution time: mean = 25.731 us, total = 845.722 ms, Queueing time: mean = 9.913 us, max = 6.895 ms, min = 1.268 us, total = 325.808 ms GcsInMemoryStore.Put - 13191 total (0 active), Execution time: mean = 22.392 us, total = 295.368 ms, Queueing time: mean = 163.373 us, max = 225.278 ms, min = 1.495 us, total = 2.155 s RayletLoadPulled - 8218 total (1 active), Execution time: mean = 244.287 us, total = 2.008 s, Queueing time: mean = 73.768 us, max = 1.351 ms, min = 10.621 us, total = 606.223 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8216 total (0 active), Execution time: mean = 75.016 us, total = 616.328 ms, Queueing time: mean = 99.495 us, max = 15.090 ms, min = 11.600 us, total = 817.453 ms NodeManagerService.grpc_client.GetResourceLoad - 8216 total (0 active), Execution time: mean = 1.364 ms, total = 11.208 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3295 total (0 active), Execution time: mean = 491.480 us, total = 1.619 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3295 total (0 active), Execution time: mean = 39.880 us, total = 131.403 ms, Queueing time: mean = 90.414 us, max = 601.063 us, min = 9.986 us, total = 297.913 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3287 total (0 active), Execution time: mean = 35.035 us, total = 115.161 ms, Queueing time: mean = 105.732 us, max = 1.496 ms, min = 14.438 us, total = 347.542 ms NodeInfoGcsService.grpc_server.CheckAlive - 3287 total (0 active), Execution time: mean = 512.993 us, total = 1.686 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2740 total (1 active), Execution time: mean = 10.196 us, total = 27.937 ms, Queueing time: mean = 74.319 us, max = 1.648 ms, min = 7.259 us, total = 203.633 ms HealthCheck - 2738 total (0 active), Execution time: mean = 7.609 us, total = 20.833 ms, Queueing time: mean = 90.721 us, max = 1.139 ms, min = 11.335 us, total = 248.395 ms GcsInMemoryStore.Get - 1700 total (0 active), Execution time: mean = 22.261 us, total = 37.843 ms, Queueing time: mean = 12.442 us, max = 137.899 us, min = 2.227 us, total = 21.151 ms InternalKVGcsService.grpc_server.InternalKVGet - 1699 total (0 active), Execution time: mean = 566.768 us, total = 962.939 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1699 total (0 active), Execution time: mean = 55.211 us, total = 93.804 ms, Queueing time: mean = 132.821 us, max = 22.982 ms, min = 9.184 us, total = 225.664 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1640 total (0 active), Execution time: mean = 616.087 us, total = 1.010 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1640 total (0 active), Execution time: mean = 90.893 us, total = 149.064 ms, Queueing time: mean = 105.045 us, max = 273.843 us, min = 15.511 us, total = 172.274 ms GCSServer.deadline_timer.debug_state_dump - 822 total (1 active), Execution time: mean = 1.361 ms, total = 1.118 s, Queueing time: mean = 39.779 us, max = 178.374 us, min = -0.023 s, total = 32.699 ms GCSServer.deadline_timer.debug_state_event_stats_print - 137 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 208.140 ms, Queueing time: mean = 58.935 us, max = 238.837 us, min = 13.943 us, total = 8.074 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:29:15,405 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 379587 total (1 active) Queueing time: mean = 40.244 us, max = 58.960 ms, min = 1.891 us, total = 15.276 s Execution time: mean = 215.121 us, total = 81.657 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 172536 total (0 active), Execution time: mean = 445.416 us, total = 76.850 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 172536 total (0 active), Execution time: mean = 22.871 us, total = 3.946 s, Queueing time: mean = 86.138 us, max = 58.960 ms, min = 2.005 us, total = 14.862 s event_loop_lag_probe - 32869 total (0 active), Execution time: mean = 25.709 us, total = 845.020 ms, Queueing time: mean = 8.953 us, max = 395.058 us, min = 1.891 us, total = 294.266 ms GcsTaskManager.GcJobSummary - 1645 total (1 active), Execution time: mean = 9.534 us, total = 15.684 ms, Queueing time: mean = 72.943 us, max = 3.642 ms, min = 12.167 us, total = 119.991 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:29:15,405 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40432 total (25 active) Queueing time: mean = 15.613 us, max = 1.217 ms, min = 1.567 us, total = 631.280 ms Execution time: mean = 4.676 s, total = 189068.665 s Event stats: event_loop_lag_probe - 32869 total (0 active), Execution time: mean = 25.565 us, total = 840.284 ms, Queueing time: mean = 9.260 us, max = 1.017 ms, min = 1.567 us, total = 304.360 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2035 total (0 active), Execution time: mean = 36.554 us, total = 74.388 ms, Queueing time: mean = 83.262 us, max = 1.217 ms, min = 4.084 us, total = 169.437 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2035 total (24 active), Execution time: mean = 92.907 s, total = 189066.590 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1707 total (0 active), Execution time: mean = 102.448 us, total = 174.879 ms, Queueing time: mean = 89.975 us, max = 260.819 us, min = 11.233 us, total = 153.588 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1707 total (0 active), Execution time: mean = 549.142 us, total = 937.385 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 439.954 us, total = 12.319 ms, Queueing time: mean = 68.003 us, max = 114.456 us, min = 33.142 us, total = 1.904 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:29:15,405 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32916 total (0 active) Queueing time: mean = 9.795 us, max = 2.505 ms, min = 563.000 ns, total = 322.412 ms Execution time: mean = 25.803 us, total = 849.341 ms Event stats: event_loop_lag_probe - 32870 total (0 active), Execution time: mean = 25.727 us, total = 845.656 ms, Queueing time: mean = 9.731 us, max = 2.505 ms, min = 1.873 us, total = 319.856 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:30:15,406 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1652 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1719 - cumulative published bytes: 1365102 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:30:15,407 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 99558 total (4 active) Queueing time: mean = 60.303 us, max = 225.583 ms, min = -0.023 s, total = 6.004 s Execution time: mean = 225.686 us, total = 22.469 s Event stats: event_loop_lag_probe - 33108 total (0 active), Execution time: mean = 25.717 us, total = 851.435 ms, Queueing time: mean = 9.907 us, max = 6.895 ms, min = 1.268 us, total = 328.013 ms GcsInMemoryStore.Put - 13287 total (0 active), Execution time: mean = 22.274 us, total = 295.961 ms, Queueing time: mean = 162.492 us, max = 225.278 ms, min = 1.495 us, total = 2.159 s RayletLoadPulled - 8278 total (1 active), Execution time: mean = 244.368 us, total = 2.023 s, Queueing time: mean = 73.753 us, max = 1.351 ms, min = 10.621 us, total = 610.528 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8276 total (0 active), Execution time: mean = 75.034 us, total = 620.978 ms, Queueing time: mean = 99.583 us, max = 15.090 ms, min = 11.600 us, total = 824.149 ms NodeManagerService.grpc_client.GetResourceLoad - 8276 total (0 active), Execution time: mean = 1.365 ms, total = 11.299 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3319 total (0 active), Execution time: mean = 491.948 us, total = 1.633 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3319 total (0 active), Execution time: mean = 39.884 us, total = 132.376 ms, Queueing time: mean = 90.474 us, max = 601.063 us, min = 9.986 us, total = 300.282 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3311 total (0 active), Execution time: mean = 35.051 us, total = 116.055 ms, Queueing time: mean = 105.739 us, max = 1.496 ms, min = 14.438 us, total = 350.102 ms NodeInfoGcsService.grpc_server.CheckAlive - 3311 total (0 active), Execution time: mean = 513.353 us, total = 1.700 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2760 total (1 active), Execution time: mean = 10.193 us, total = 28.132 ms, Queueing time: mean = 74.344 us, max = 1.648 ms, min = 7.259 us, total = 205.190 ms HealthCheck - 2758 total (0 active), Execution time: mean = 7.612 us, total = 20.993 ms, Queueing time: mean = 90.827 us, max = 1.139 ms, min = 11.335 us, total = 250.502 ms GcsInMemoryStore.Get - 1712 total (0 active), Execution time: mean = 22.276 us, total = 38.136 ms, Queueing time: mean = 12.438 us, max = 137.899 us, min = 2.227 us, total = 21.293 ms InternalKVGcsService.grpc_server.InternalKVGet - 1711 total (0 active), Execution time: mean = 566.830 us, total = 969.847 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1711 total (0 active), Execution time: mean = 55.189 us, total = 94.428 ms, Queueing time: mean = 132.610 us, max = 22.982 ms, min = 9.184 us, total = 226.895 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1652 total (0 active), Execution time: mean = 616.272 us, total = 1.018 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1652 total (0 active), Execution time: mean = 90.869 us, total = 150.115 ms, Queueing time: mean = 105.082 us, max = 273.843 us, min = 15.511 us, total = 173.595 ms GCSServer.deadline_timer.debug_state_dump - 828 total (1 active), Execution time: mean = 1.361 ms, total = 1.127 s, Queueing time: mean = 39.994 us, max = 178.374 us, min = -0.023 s, total = 33.115 ms GCSServer.deadline_timer.debug_state_event_stats_print - 138 total (1 active, 1 running), Execution time: mean = 1.520 ms, total = 209.811 ms, Queueing time: mean = 58.979 us, max = 238.837 us, min = 13.943 us, total = 8.139 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:30:15,407 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 382359 total (1 active) Queueing time: mean = 40.318 us, max = 58.960 ms, min = 1.891 us, total = 15.416 s Execution time: mean = 215.448 us, total = 82.379 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 173796 total (0 active), Execution time: mean = 446.114 us, total = 77.533 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 173796 total (0 active), Execution time: mean = 22.896 us, total = 3.979 s, Queueing time: mean = 86.302 us, max = 58.960 ms, min = 2.005 us, total = 14.999 s event_loop_lag_probe - 33109 total (0 active), Execution time: mean = 25.695 us, total = 850.749 ms, Queueing time: mean = 8.948 us, max = 395.058 us, min = 1.891 us, total = 296.275 ms GcsTaskManager.GcJobSummary - 1657 total (1 active), Execution time: mean = 9.526 us, total = 15.784 ms, Queueing time: mean = 72.907 us, max = 3.642 ms, min = 12.167 us, total = 120.807 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:30:15,407 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40720 total (25 active) Queueing time: mean = 15.623 us, max = 1.217 ms, min = 1.567 us, total = 636.168 ms Execution time: mean = 4.645 s, total = 189128.801 s Event stats: event_loop_lag_probe - 33109 total (0 active), Execution time: mean = 25.566 us, total = 846.473 ms, Queueing time: mean = 9.260 us, max = 1.017 ms, min = 1.567 us, total = 306.580 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2047 total (0 active), Execution time: mean = 36.565 us, total = 74.849 ms, Queueing time: mean = 83.433 us, max = 1.217 ms, min = 4.084 us, total = 170.788 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2047 total (24 active), Execution time: mean = 92.392 s, total = 189126.710 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1719 total (0 active), Execution time: mean = 102.375 us, total = 175.983 ms, Queueing time: mean = 90.113 us, max = 260.819 us, min = 11.233 us, total = 154.904 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1719 total (0 active), Execution time: mean = 549.763 us, total = 945.042 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 439.954 us, total = 12.319 ms, Queueing time: mean = 68.003 us, max = 114.456 us, min = 33.142 us, total = 1.904 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:30:15,407 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33156 total (0 active) Queueing time: mean = 9.793 us, max = 2.505 ms, min = 563.000 ns, total = 324.704 ms Execution time: mean = 25.797 us, total = 855.341 ms Event stats: event_loop_lag_probe - 33110 total (0 active), Execution time: mean = 25.722 us, total = 851.657 ms, Queueing time: mean = 9.730 us, max = 2.505 ms, min = 1.873 us, total = 322.148 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:31:15,407 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1664 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1731 - cumulative published bytes: 1374930 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:31:15,408 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 100277 total (4 active) Queueing time: mean = 60.106 us, max = 225.583 ms, min = -0.023 s, total = 6.027 s Execution time: mean = 225.657 us, total = 22.628 s Event stats: event_loop_lag_probe - 33348 total (0 active), Execution time: mean = 25.698 us, total = 856.985 ms, Queueing time: mean = 9.900 us, max = 6.895 ms, min = 1.268 us, total = 330.130 ms GcsInMemoryStore.Put - 13383 total (0 active), Execution time: mean = 22.152 us, total = 296.463 ms, Queueing time: mean = 161.606 us, max = 225.278 ms, min = 1.495 us, total = 2.163 s RayletLoadPulled - 8338 total (1 active), Execution time: mean = 244.294 us, total = 2.037 s, Queueing time: mean = 73.656 us, max = 1.351 ms, min = 10.621 us, total = 614.140 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8336 total (0 active), Execution time: mean = 74.983 us, total = 625.057 ms, Queueing time: mean = 99.417 us, max = 15.090 ms, min = 11.600 us, total = 828.740 ms NodeManagerService.grpc_client.GetResourceLoad - 8336 total (0 active), Execution time: mean = 1.366 ms, total = 11.384 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3343 total (0 active), Execution time: mean = 491.891 us, total = 1.644 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3343 total (0 active), Execution time: mean = 39.847 us, total = 133.210 ms, Queueing time: mean = 90.360 us, max = 601.063 us, min = 9.986 us, total = 302.074 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3335 total (0 active), Execution time: mean = 35.035 us, total = 116.843 ms, Queueing time: mean = 105.617 us, max = 1.496 ms, min = 14.438 us, total = 352.233 ms NodeInfoGcsService.grpc_server.CheckAlive - 3335 total (0 active), Execution time: mean = 513.061 us, total = 1.711 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2780 total (1 active), Execution time: mean = 10.179 us, total = 28.296 ms, Queueing time: mean = 74.218 us, max = 1.648 ms, min = 7.259 us, total = 206.325 ms HealthCheck - 2778 total (0 active), Execution time: mean = 7.616 us, total = 21.157 ms, Queueing time: mean = 90.799 us, max = 1.139 ms, min = 11.335 us, total = 252.241 ms GcsInMemoryStore.Get - 1724 total (0 active), Execution time: mean = 22.286 us, total = 38.420 ms, Queueing time: mean = 12.431 us, max = 137.899 us, min = 2.227 us, total = 21.432 ms InternalKVGcsService.grpc_server.InternalKVGet - 1723 total (0 active), Execution time: mean = 566.665 us, total = 976.363 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1723 total (0 active), Execution time: mean = 55.130 us, total = 94.989 ms, Queueing time: mean = 132.289 us, max = 22.982 ms, min = 9.184 us, total = 227.934 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1664 total (0 active), Execution time: mean = 616.219 us, total = 1.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1664 total (0 active), Execution time: mean = 90.868 us, total = 151.204 ms, Queueing time: mean = 105.026 us, max = 273.843 us, min = 15.511 us, total = 174.763 ms GCSServer.deadline_timer.debug_state_dump - 834 total (1 active), Execution time: mean = 1.361 ms, total = 1.135 s, Queueing time: mean = 40.190 us, max = 178.374 us, min = -0.023 s, total = 33.519 ms GCSServer.deadline_timer.debug_state_event_stats_print - 139 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 211.383 ms, Queueing time: mean = 59.017 us, max = 238.837 us, min = 13.943 us, total = 8.203 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:31:15,408 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 385131 total (1 active) Queueing time: mean = 40.306 us, max = 58.960 ms, min = 1.891 us, total = 15.523 s Execution time: mean = 215.440 us, total = 82.972 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 175056 total (0 active), Execution time: mean = 446.100 us, total = 78.092 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 175056 total (0 active), Execution time: mean = 22.896 us, total = 4.008 s, Queueing time: mean = 86.278 us, max = 58.960 ms, min = 2.005 us, total = 15.103 s event_loop_lag_probe - 33349 total (0 active), Execution time: mean = 25.668 us, total = 856.017 ms, Queueing time: mean = 8.938 us, max = 395.058 us, min = 1.891 us, total = 298.088 ms GcsTaskManager.GcJobSummary - 1669 total (1 active), Execution time: mean = 9.539 us, total = 15.921 ms, Queueing time: mean = 72.870 us, max = 3.642 ms, min = 12.167 us, total = 121.621 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:31:15,409 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41008 total (25 active) Queueing time: mean = 15.613 us, max = 1.217 ms, min = 1.567 us, total = 640.272 ms Execution time: mean = 4.613 s, total = 189189.026 s Event stats: event_loop_lag_probe - 33349 total (0 active), Execution time: mean = 25.559 us, total = 852.379 ms, Queueing time: mean = 9.259 us, max = 1.017 ms, min = 1.567 us, total = 308.764 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2059 total (0 active), Execution time: mean = 36.592 us, total = 75.342 ms, Queueing time: mean = 83.315 us, max = 1.217 ms, min = 4.084 us, total = 171.546 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2059 total (24 active), Execution time: mean = 91.883 s, total = 189186.920 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1731 total (0 active), Execution time: mean = 102.384 us, total = 177.227 ms, Queueing time: mean = 90.160 us, max = 260.819 us, min = 11.233 us, total = 156.067 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1731 total (0 active), Execution time: mean = 549.840 us, total = 951.772 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 439.954 us, total = 12.319 ms, Queueing time: mean = 68.003 us, max = 114.456 us, min = 33.142 us, total = 1.904 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:31:15,409 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33396 total (0 active) Queueing time: mean = 9.785 us, max = 2.505 ms, min = 563.000 ns, total = 326.785 ms Execution time: mean = 25.778 us, total = 860.867 ms Event stats: event_loop_lag_probe - 33350 total (0 active), Execution time: mean = 25.703 us, total = 857.183 ms, Queueing time: mean = 9.722 us, max = 2.505 ms, min = 1.873 us, total = 324.229 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:32:15,409 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1676 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1743 - cumulative published bytes: 1384758 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:32:15,410 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 100994 total (4 active) Queueing time: mean = 59.924 us, max = 225.583 ms, min = -0.023 s, total = 6.052 s Execution time: mean = 225.570 us, total = 22.781 s Event stats: event_loop_lag_probe - 33588 total (0 active), Execution time: mean = 25.678 us, total = 862.474 ms, Queueing time: mean = 9.892 us, max = 6.895 ms, min = 1.268 us, total = 332.248 ms GcsInMemoryStore.Put - 13479 total (0 active), Execution time: mean = 22.033 us, total = 296.987 ms, Queueing time: mean = 160.748 us, max = 225.278 ms, min = 1.495 us, total = 2.167 s RayletLoadPulled - 8398 total (1 active), Execution time: mean = 244.248 us, total = 2.051 s, Queueing time: mean = 73.575 us, max = 1.351 ms, min = 10.621 us, total = 617.880 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8396 total (0 active), Execution time: mean = 74.970 us, total = 629.451 ms, Queueing time: mean = 99.350 us, max = 15.090 ms, min = 11.600 us, total = 834.144 ms NodeManagerService.grpc_client.GetResourceLoad - 8396 total (0 active), Execution time: mean = 1.365 ms, total = 11.464 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3367 total (0 active), Execution time: mean = 491.751 us, total = 1.656 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3367 total (0 active), Execution time: mean = 39.788 us, total = 133.967 ms, Queueing time: mean = 90.286 us, max = 601.063 us, min = 9.911 us, total = 303.993 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3358 total (0 active), Execution time: mean = 35.021 us, total = 117.599 ms, Queueing time: mean = 105.444 us, max = 1.496 ms, min = 14.438 us, total = 354.080 ms NodeInfoGcsService.grpc_server.CheckAlive - 3358 total (0 active), Execution time: mean = 512.780 us, total = 1.722 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2800 total (1 active), Execution time: mean = 10.165 us, total = 28.463 ms, Queueing time: mean = 74.194 us, max = 1.648 ms, min = 7.259 us, total = 207.742 ms HealthCheck - 2798 total (0 active), Execution time: mean = 7.613 us, total = 21.301 ms, Queueing time: mean = 90.734 us, max = 1.139 ms, min = 11.335 us, total = 253.875 ms GcsInMemoryStore.Get - 1736 total (0 active), Execution time: mean = 22.302 us, total = 38.716 ms, Queueing time: mean = 12.426 us, max = 137.899 us, min = 2.227 us, total = 21.572 ms InternalKVGcsService.grpc_server.InternalKVGet - 1735 total (0 active), Execution time: mean = 566.307 us, total = 982.542 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1735 total (0 active), Execution time: mean = 55.072 us, total = 95.551 ms, Queueing time: mean = 132.001 us, max = 22.982 ms, min = 9.184 us, total = 229.022 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1676 total (0 active), Execution time: mean = 616.051 us, total = 1.033 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1676 total (0 active), Execution time: mean = 90.823 us, total = 152.220 ms, Queueing time: mean = 104.895 us, max = 273.843 us, min = 15.511 us, total = 175.804 ms GCSServer.deadline_timer.debug_state_dump - 840 total (1 active), Execution time: mean = 1.361 ms, total = 1.143 s, Queueing time: mean = 40.274 us, max = 178.374 us, min = -0.023 s, total = 33.830 ms GCSServer.deadline_timer.debug_state_event_stats_print - 140 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 212.991 ms, Queueing time: mean = 58.861 us, max = 238.837 us, min = 13.943 us, total = 8.241 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:32:15,410 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 387903 total (1 active) Queueing time: mean = 40.300 us, max = 58.960 ms, min = 1.891 us, total = 15.632 s Execution time: mean = 215.434 us, total = 83.568 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 176316 total (0 active), Execution time: mean = 446.090 us, total = 78.653 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 176316 total (0 active), Execution time: mean = 22.897 us, total = 4.037 s, Queueing time: mean = 86.265 us, max = 58.960 ms, min = 2.005 us, total = 15.210 s event_loop_lag_probe - 33589 total (0 active), Execution time: mean = 25.653 us, total = 861.652 ms, Queueing time: mean = 8.934 us, max = 395.058 us, min = 1.891 us, total = 300.097 ms GcsTaskManager.GcJobSummary - 1681 total (1 active), Execution time: mean = 9.544 us, total = 16.044 ms, Queueing time: mean = 72.777 us, max = 3.642 ms, min = 12.167 us, total = 122.338 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:32:15,410 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41333 total (25 active) Queueing time: mean = 15.638 us, max = 1.217 ms, min = 1.567 us, total = 646.382 ms Execution time: mean = 4.840 s, total = 200049.163 s Event stats: event_loop_lag_probe - 33589 total (0 active), Execution time: mean = 25.557 us, total = 858.419 ms, Queueing time: mean = 9.257 us, max = 1.017 ms, min = 1.567 us, total = 310.927 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2089 total (0 active), Execution time: mean = 36.504 us, total = 76.257 ms, Queueing time: mean = 83.455 us, max = 1.217 ms, min = 4.084 us, total = 174.337 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2089 total (24 active), Execution time: mean = 95.762 s, total = 200047.042 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1743 total (0 active), Execution time: mean = 102.367 us, total = 178.426 ms, Queueing time: mean = 90.153 us, max = 260.819 us, min = 11.233 us, total = 157.137 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1743 total (0 active), Execution time: mean = 549.849 us, total = 958.387 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 443.052 us, total = 12.849 ms, Queueing time: mean = 68.588 us, max = 114.456 us, min = 33.142 us, total = 1.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:32:15,410 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33636 total (0 active) Queueing time: mean = 9.778 us, max = 2.505 ms, min = 563.000 ns, total = 328.900 ms Execution time: mean = 25.759 us, total = 866.436 ms Event stats: event_loop_lag_probe - 33590 total (0 active), Execution time: mean = 25.685 us, total = 862.752 ms, Queueing time: mean = 9.715 us, max = 2.505 ms, min = 1.873 us, total = 326.343 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:33:15,411 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1688 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1756 - cumulative published bytes: 1394620 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:33:15,412 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 101713 total (4 active) Queueing time: mean = 59.793 us, max = 225.583 ms, min = -0.023 s, total = 6.082 s Execution time: mean = 225.691 us, total = 22.956 s Event stats: event_loop_lag_probe - 33828 total (0 active), Execution time: mean = 25.667 us, total = 868.247 ms, Queueing time: mean = 9.888 us, max = 6.895 ms, min = 1.268 us, total = 334.484 ms GcsInMemoryStore.Put - 13575 total (0 active), Execution time: mean = 21.923 us, total = 297.600 ms, Queueing time: mean = 159.896 us, max = 225.278 ms, min = 1.495 us, total = 2.171 s RayletLoadPulled - 8458 total (1 active), Execution time: mean = 244.249 us, total = 2.066 s, Queueing time: mean = 73.553 us, max = 1.351 ms, min = 10.621 us, total = 622.110 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8456 total (0 active), Execution time: mean = 74.984 us, total = 634.065 ms, Queueing time: mean = 99.440 us, max = 15.090 ms, min = 11.600 us, total = 840.864 ms NodeManagerService.grpc_client.GetResourceLoad - 8456 total (0 active), Execution time: mean = 1.367 ms, total = 11.556 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3391 total (0 active), Execution time: mean = 492.314 us, total = 1.669 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3391 total (0 active), Execution time: mean = 39.767 us, total = 134.849 ms, Queueing time: mean = 90.468 us, max = 601.063 us, min = 9.911 us, total = 306.777 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3382 total (0 active), Execution time: mean = 35.039 us, total = 118.503 ms, Queueing time: mean = 105.483 us, max = 1.496 ms, min = 14.438 us, total = 356.742 ms NodeInfoGcsService.grpc_server.CheckAlive - 3382 total (0 active), Execution time: mean = 513.177 us, total = 1.736 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2820 total (1 active), Execution time: mean = 10.160 us, total = 28.650 ms, Queueing time: mean = 74.258 us, max = 1.648 ms, min = 7.259 us, total = 209.407 ms HealthCheck - 2818 total (0 active), Execution time: mean = 7.617 us, total = 21.464 ms, Queueing time: mean = 90.845 us, max = 1.139 ms, min = 11.335 us, total = 256.000 ms GcsInMemoryStore.Get - 1748 total (0 active), Execution time: mean = 22.310 us, total = 38.999 ms, Queueing time: mean = 12.438 us, max = 137.899 us, min = 2.227 us, total = 21.741 ms InternalKVGcsService.grpc_server.InternalKVGet - 1747 total (0 active), Execution time: mean = 566.560 us, total = 989.780 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1747 total (0 active), Execution time: mean = 55.050 us, total = 96.172 ms, Queueing time: mean = 131.857 us, max = 22.982 ms, min = 9.184 us, total = 230.354 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1688 total (0 active), Execution time: mean = 616.406 us, total = 1.040 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1688 total (0 active), Execution time: mean = 90.814 us, total = 153.294 ms, Queueing time: mean = 105.000 us, max = 273.843 us, min = 15.511 us, total = 177.240 ms GCSServer.deadline_timer.debug_state_dump - 846 total (1 active), Execution time: mean = 1.362 ms, total = 1.152 s, Queueing time: mean = 40.548 us, max = 178.374 us, min = -0.023 s, total = 34.304 ms GCSServer.deadline_timer.debug_state_event_stats_print - 141 total (1 active, 1 running), Execution time: mean = 1.522 ms, total = 214.587 ms, Queueing time: mean = 59.065 us, max = 238.837 us, min = 13.943 us, total = 8.328 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:33:15,412 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 390674 total (1 active) Queueing time: mean = 40.386 us, max = 58.960 ms, min = 1.891 us, total = 15.778 s Execution time: mean = 215.761 us, total = 84.292 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 177576 total (0 active), Execution time: mean = 446.779 us, total = 79.337 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 177576 total (0 active), Execution time: mean = 22.927 us, total = 4.071 s, Queueing time: mean = 86.455 us, max = 58.960 ms, min = 2.005 us, total = 15.352 s event_loop_lag_probe - 33828 total (0 active), Execution time: mean = 25.645 us, total = 867.514 ms, Queueing time: mean = 8.932 us, max = 395.058 us, min = 1.891 us, total = 302.141 ms GcsTaskManager.GcJobSummary - 1693 total (1 active), Execution time: mean = 9.539 us, total = 16.150 ms, Queueing time: mean = 72.788 us, max = 3.642 ms, min = 12.167 us, total = 123.230 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:33:15,412 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41625 total (25 active) Queueing time: mean = 15.662 us, max = 1.217 ms, min = 1.567 us, total = 651.910 ms Execution time: mean = 4.807 s, total = 200109.326 s Event stats: event_loop_lag_probe - 33829 total (0 active), Execution time: mean = 25.559 us, total = 864.651 ms, Queueing time: mean = 9.259 us, max = 1.017 ms, min = 1.567 us, total = 313.214 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2102 total (0 active), Execution time: mean = 36.584 us, total = 76.899 ms, Queueing time: mean = 83.703 us, max = 1.217 ms, min = 4.084 us, total = 175.944 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2102 total (24 active), Execution time: mean = 95.198 s, total = 200107.189 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1756 total (0 active), Execution time: mean = 102.369 us, total = 179.759 ms, Queueing time: mean = 90.417 us, max = 260.819 us, min = 11.233 us, total = 158.772 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1756 total (0 active), Execution time: mean = 550.592 us, total = 966.840 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 443.052 us, total = 12.849 ms, Queueing time: mean = 68.588 us, max = 114.456 us, min = 33.142 us, total = 1.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:33:15,412 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33876 total (0 active) Queueing time: mean = 9.777 us, max = 2.505 ms, min = 563.000 ns, total = 331.190 ms Execution time: mean = 25.756 us, total = 872.525 ms Event stats: event_loop_lag_probe - 33830 total (0 active), Execution time: mean = 25.683 us, total = 868.841 ms, Queueing time: mean = 9.714 us, max = 2.505 ms, min = 1.873 us, total = 328.634 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:34:15,412 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1700 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1768 - cumulative published bytes: 1404448 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:34:15,413 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 102432 total (4 active) Queueing time: mean = 59.664 us, max = 225.583 ms, min = -0.023 s, total = 6.111 s Execution time: mean = 225.891 us, total = 23.138 s Event stats: event_loop_lag_probe - 34068 total (0 active), Execution time: mean = 25.653 us, total = 873.957 ms, Queueing time: mean = 9.881 us, max = 6.895 ms, min = 1.268 us, total = 336.640 ms GcsInMemoryStore.Put - 13671 total (0 active), Execution time: mean = 21.815 us, total = 298.229 ms, Queueing time: mean = 159.080 us, max = 225.278 ms, min = 1.495 us, total = 2.175 s RayletLoadPulled - 8518 total (1 active), Execution time: mean = 244.369 us, total = 2.082 s, Queueing time: mean = 73.531 us, max = 1.351 ms, min = 10.621 us, total = 626.341 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8516 total (0 active), Execution time: mean = 74.997 us, total = 638.672 ms, Queueing time: mean = 99.541 us, max = 15.090 ms, min = 11.600 us, total = 847.694 ms NodeManagerService.grpc_client.GetResourceLoad - 8516 total (0 active), Execution time: mean = 1.369 ms, total = 11.654 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3415 total (0 active), Execution time: mean = 492.833 us, total = 1.683 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3415 total (0 active), Execution time: mean = 39.809 us, total = 135.947 ms, Queueing time: mean = 90.585 us, max = 601.063 us, min = 9.911 us, total = 309.349 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3406 total (0 active), Execution time: mean = 35.054 us, total = 119.394 ms, Queueing time: mean = 105.517 us, max = 1.496 ms, min = 14.438 us, total = 359.390 ms NodeInfoGcsService.grpc_server.CheckAlive - 3406 total (0 active), Execution time: mean = 513.558 us, total = 1.749 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2840 total (1 active), Execution time: mean = 10.152 us, total = 28.833 ms, Queueing time: mean = 74.306 us, max = 1.648 ms, min = 7.259 us, total = 211.029 ms HealthCheck - 2838 total (0 active), Execution time: mean = 7.622 us, total = 21.632 ms, Queueing time: mean = 90.925 us, max = 1.139 ms, min = 11.335 us, total = 258.046 ms GcsInMemoryStore.Get - 1760 total (0 active), Execution time: mean = 22.344 us, total = 39.325 ms, Queueing time: mean = 12.452 us, max = 137.899 us, min = 2.227 us, total = 21.915 ms InternalKVGcsService.grpc_server.InternalKVGet - 1759 total (0 active), Execution time: mean = 566.835 us, total = 997.063 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1759 total (0 active), Execution time: mean = 55.097 us, total = 96.915 ms, Queueing time: mean = 131.680 us, max = 22.982 ms, min = 9.184 us, total = 231.625 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1700 total (0 active), Execution time: mean = 616.714 us, total = 1.048 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1700 total (0 active), Execution time: mean = 90.847 us, total = 154.440 ms, Queueing time: mean = 105.086 us, max = 273.843 us, min = 15.511 us, total = 178.646 ms GCSServer.deadline_timer.debug_state_dump - 852 total (1 active), Execution time: mean = 1.362 ms, total = 1.161 s, Queueing time: mean = 40.913 us, max = 178.374 us, min = -0.023 s, total = 34.858 ms GCSServer.deadline_timer.debug_state_event_stats_print - 142 total (1 active, 1 running), Execution time: mean = 1.522 ms, total = 216.169 ms, Queueing time: mean = 59.237 us, max = 238.837 us, min = 13.943 us, total = 8.412 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:34:15,413 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 393446 total (1 active) Queueing time: mean = 40.460 us, max = 58.960 ms, min = 1.891 us, total = 15.919 s Execution time: mean = 216.089 us, total = 85.020 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 178836 total (0 active), Execution time: mean = 447.474 us, total = 80.024 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 178836 total (0 active), Execution time: mean = 22.957 us, total = 4.105 s, Queueing time: mean = 86.619 us, max = 58.960 ms, min = 2.005 us, total = 15.491 s event_loop_lag_probe - 34068 total (0 active), Execution time: mean = 25.636 us, total = 873.382 ms, Queueing time: mean = 8.930 us, max = 395.058 us, min = 1.891 us, total = 304.225 ms GcsTaskManager.GcJobSummary - 1705 total (1 active), Execution time: mean = 9.541 us, total = 16.267 ms, Queueing time: mean = 72.814 us, max = 3.642 ms, min = 12.167 us, total = 124.149 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:34:15,413 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41913 total (25 active) Queueing time: mean = 15.672 us, max = 1.217 ms, min = 1.567 us, total = 656.877 ms Execution time: mean = 4.776 s, total = 200169.549 s Event stats: event_loop_lag_probe - 34069 total (0 active), Execution time: mean = 25.562 us, total = 870.870 ms, Queueing time: mean = 9.259 us, max = 1.017 ms, min = 1.567 us, total = 315.435 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2114 total (0 active), Execution time: mean = 36.642 us, total = 77.462 ms, Queueing time: mean = 83.877 us, max = 1.217 ms, min = 4.084 us, total = 177.315 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2114 total (24 active), Execution time: mean = 94.687 s, total = 200167.396 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1768 total (0 active), Execution time: mean = 102.471 us, total = 181.168 ms, Queueing time: mean = 90.581 us, max = 260.819 us, min = 11.233 us, total = 160.147 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1768 total (0 active), Execution time: mean = 551.328 us, total = 974.748 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 443.052 us, total = 12.849 ms, Queueing time: mean = 68.588 us, max = 114.456 us, min = 33.142 us, total = 1.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:34:15,413 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34115 total (0 active) Queueing time: mean = 9.772 us, max = 2.505 ms, min = 563.000 ns, total = 333.384 ms Execution time: mean = 25.747 us, total = 878.367 ms Event stats: event_loop_lag_probe - 34069 total (0 active), Execution time: mean = 25.674 us, total = 874.683 ms, Queueing time: mean = 9.711 us, max = 2.505 ms, min = 1.873 us, total = 330.828 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:35:15,413 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1712 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1780 - cumulative published bytes: 1414276 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:35:15,414 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 103151 total (4 active) Queueing time: mean = 59.543 us, max = 225.583 ms, min = -0.023 s, total = 6.142 s Execution time: mean = 226.019 us, total = 23.314 s Event stats: event_loop_lag_probe - 34308 total (0 active), Execution time: mean = 25.637 us, total = 879.555 ms, Queueing time: mean = 9.874 us, max = 6.895 ms, min = 1.268 us, total = 338.746 ms GcsInMemoryStore.Put - 13767 total (0 active), Execution time: mean = 21.709 us, total = 298.874 ms, Queueing time: mean = 158.291 us, max = 225.278 ms, min = 1.495 us, total = 2.179 s RayletLoadPulled - 8578 total (1 active), Execution time: mean = 244.450 us, total = 2.097 s, Queueing time: mean = 73.549 us, max = 1.351 ms, min = 10.621 us, total = 630.906 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8576 total (0 active), Execution time: mean = 75.032 us, total = 643.472 ms, Queueing time: mean = 99.683 us, max = 15.090 ms, min = 11.600 us, total = 854.886 ms NodeManagerService.grpc_client.GetResourceLoad - 8576 total (0 active), Execution time: mean = 1.370 ms, total = 11.747 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3439 total (0 active), Execution time: mean = 493.484 us, total = 1.697 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3439 total (0 active), Execution time: mean = 39.803 us, total = 136.882 ms, Queueing time: mean = 90.728 us, max = 601.063 us, min = 9.911 us, total = 312.013 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3430 total (0 active), Execution time: mean = 35.063 us, total = 120.265 ms, Queueing time: mean = 105.602 us, max = 1.496 ms, min = 14.438 us, total = 362.214 ms NodeInfoGcsService.grpc_server.CheckAlive - 3430 total (0 active), Execution time: mean = 513.922 us, total = 1.763 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2860 total (1 active), Execution time: mean = 10.155 us, total = 29.042 ms, Queueing time: mean = 74.297 us, max = 1.648 ms, min = 7.259 us, total = 212.490 ms HealthCheck - 2858 total (0 active), Execution time: mean = 7.627 us, total = 21.798 ms, Queueing time: mean = 90.955 us, max = 1.139 ms, min = 11.335 us, total = 259.950 ms GcsInMemoryStore.Get - 1772 total (0 active), Execution time: mean = 22.368 us, total = 39.636 ms, Queueing time: mean = 12.456 us, max = 137.899 us, min = 2.227 us, total = 22.072 ms InternalKVGcsService.grpc_server.InternalKVGet - 1771 total (0 active), Execution time: mean = 567.005 us, total = 1.004 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1771 total (0 active), Execution time: mean = 55.095 us, total = 97.573 ms, Queueing time: mean = 131.559 us, max = 22.982 ms, min = 9.184 us, total = 232.992 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1712 total (0 active), Execution time: mean = 616.910 us, total = 1.056 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1712 total (0 active), Execution time: mean = 90.832 us, total = 155.505 ms, Queueing time: mean = 105.141 us, max = 273.843 us, min = 15.511 us, total = 180.002 ms GCSServer.deadline_timer.debug_state_dump - 858 total (1 active), Execution time: mean = 1.363 ms, total = 1.169 s, Queueing time: mean = 41.090 us, max = 178.374 us, min = -0.023 s, total = 35.255 ms GCSServer.deadline_timer.debug_state_event_stats_print - 143 total (1 active, 1 running), Execution time: mean = 1.519 ms, total = 217.230 ms, Queueing time: mean = 59.303 us, max = 238.837 us, min = 13.943 us, total = 8.480 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:35:15,415 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 396218 total (1 active) Queueing time: mean = 40.531 us, max = 58.960 ms, min = 1.891 us, total = 16.059 s Execution time: mean = 216.401 us, total = 85.742 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 180096 total (0 active), Execution time: mean = 448.131 us, total = 80.707 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 180096 total (0 active), Execution time: mean = 22.986 us, total = 4.140 s, Queueing time: mean = 86.775 us, max = 58.960 ms, min = 2.005 us, total = 15.628 s event_loop_lag_probe - 34308 total (0 active), Execution time: mean = 25.629 us, total = 879.278 ms, Queueing time: mean = 8.928 us, max = 395.058 us, min = 1.891 us, total = 306.297 ms GcsTaskManager.GcJobSummary - 1717 total (1 active), Execution time: mean = 9.542 us, total = 16.384 ms, Queueing time: mean = 72.806 us, max = 3.642 ms, min = 12.167 us, total = 125.008 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:35:15,415 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 42201 total (25 active) Queueing time: mean = 15.684 us, max = 1.217 ms, min = 1.567 us, total = 661.896 ms Execution time: mean = 4.745 s, total = 200229.698 s Event stats: event_loop_lag_probe - 34309 total (0 active), Execution time: mean = 25.563 us, total = 877.037 ms, Queueing time: mean = 9.258 us, max = 1.017 ms, min = 1.567 us, total = 317.632 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2126 total (0 active), Execution time: mean = 36.745 us, total = 78.120 ms, Queueing time: mean = 84.074 us, max = 1.217 ms, min = 4.084 us, total = 178.742 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2126 total (24 active), Execution time: mean = 94.180 s, total = 200227.529 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1780 total (0 active), Execution time: mean = 102.537 us, total = 182.517 ms, Queueing time: mean = 90.753 us, max = 260.819 us, min = 11.233 us, total = 161.541 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1780 total (0 active), Execution time: mean = 552.013 us, total = 982.584 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 443.052 us, total = 12.849 ms, Queueing time: mean = 68.588 us, max = 114.456 us, min = 33.142 us, total = 1.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:35:15,415 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34355 total (0 active) Queueing time: mean = 9.766 us, max = 2.505 ms, min = 563.000 ns, total = 335.528 ms Execution time: mean = 25.738 us, total = 884.218 ms Event stats: event_loop_lag_probe - 34309 total (0 active), Execution time: mean = 25.665 us, total = 880.534 ms, Queueing time: mean = 9.705 us, max = 2.505 ms, min = 1.873 us, total = 332.972 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:36:15,415 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1724 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1792 - cumulative published bytes: 1424104 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:36:15,416 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 103870 total (4 active) Queueing time: mean = 59.413 us, max = 225.583 ms, min = -0.023 s, total = 6.171 s Execution time: mean = 226.121 us, total = 23.487 s Event stats: event_loop_lag_probe - 34548 total (0 active), Execution time: mean = 25.621 us, total = 885.144 ms, Queueing time: mean = 9.868 us, max = 6.895 ms, min = 1.268 us, total = 340.910 ms GcsInMemoryStore.Put - 13863 total (0 active), Execution time: mean = 21.603 us, total = 299.482 ms, Queueing time: mean = 157.491 us, max = 225.278 ms, min = 1.495 us, total = 2.183 s RayletLoadPulled - 8638 total (1 active), Execution time: mean = 244.515 us, total = 2.112 s, Queueing time: mean = 73.577 us, max = 1.351 ms, min = 10.621 us, total = 635.559 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8636 total (0 active), Execution time: mean = 75.047 us, total = 648.108 ms, Queueing time: mean = 99.784 us, max = 15.090 ms, min = 11.600 us, total = 861.737 ms NodeManagerService.grpc_client.GetResourceLoad - 8636 total (0 active), Execution time: mean = 1.371 ms, total = 11.839 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3463 total (0 active), Execution time: mean = 493.971 us, total = 1.711 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3463 total (0 active), Execution time: mean = 39.799 us, total = 137.825 ms, Queueing time: mean = 90.836 us, max = 601.063 us, min = 9.911 us, total = 314.566 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3454 total (0 active), Execution time: mean = 35.060 us, total = 121.099 ms, Queueing time: mean = 105.637 us, max = 1.496 ms, min = 14.438 us, total = 364.871 ms NodeInfoGcsService.grpc_server.CheckAlive - 3454 total (0 active), Execution time: mean = 514.285 us, total = 1.776 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2880 total (1 active), Execution time: mean = 10.147 us, total = 29.222 ms, Queueing time: mean = 74.286 us, max = 1.648 ms, min = 7.259 us, total = 213.944 ms HealthCheck - 2878 total (0 active), Execution time: mean = 7.631 us, total = 21.961 ms, Queueing time: mean = 91.066 us, max = 1.139 ms, min = 11.335 us, total = 262.087 ms GcsInMemoryStore.Get - 1784 total (0 active), Execution time: mean = 22.382 us, total = 39.930 ms, Queueing time: mean = 12.453 us, max = 137.899 us, min = 2.227 us, total = 22.215 ms InternalKVGcsService.grpc_server.InternalKVGet - 1783 total (0 active), Execution time: mean = 566.968 us, total = 1.011 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1783 total (0 active), Execution time: mean = 55.075 us, total = 98.199 ms, Queueing time: mean = 131.228 us, max = 22.982 ms, min = 9.184 us, total = 233.979 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1724 total (0 active), Execution time: mean = 616.938 us, total = 1.064 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1724 total (0 active), Execution time: mean = 90.801 us, total = 156.541 ms, Queueing time: mean = 105.051 us, max = 273.843 us, min = 15.511 us, total = 181.108 ms GCSServer.deadline_timer.debug_state_dump - 864 total (1 active), Execution time: mean = 1.363 ms, total = 1.178 s, Queueing time: mean = 41.279 us, max = 178.374 us, min = -0.023 s, total = 35.665 ms GCSServer.deadline_timer.debug_state_event_stats_print - 144 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 219.018 ms, Queueing time: mean = 59.449 us, max = 238.837 us, min = 13.943 us, total = 8.561 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:36:15,416 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 398990 total (1 active) Queueing time: mean = 40.609 us, max = 58.960 ms, min = 1.891 us, total = 16.203 s Execution time: mean = 216.725 us, total = 86.471 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 181356 total (0 active), Execution time: mean = 448.810 us, total = 81.394 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 181356 total (0 active), Execution time: mean = 23.021 us, total = 4.175 s, Queueing time: mean = 86.947 us, max = 58.960 ms, min = 2.005 us, total = 15.768 s event_loop_lag_probe - 34548 total (0 active), Execution time: mean = 25.624 us, total = 885.265 ms, Queueing time: mean = 8.928 us, max = 395.058 us, min = 1.891 us, total = 308.431 ms GcsTaskManager.GcJobSummary - 1729 total (1 active), Execution time: mean = 9.550 us, total = 16.511 ms, Queueing time: mean = 72.789 us, max = 3.642 ms, min = 12.167 us, total = 125.853 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:36:15,417 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 42489 total (25 active) Queueing time: mean = 15.697 us, max = 1.217 ms, min = 1.567 us, total = 666.951 ms Execution time: mean = 4.714 s, total = 200289.945 s Event stats: event_loop_lag_probe - 34549 total (0 active), Execution time: mean = 25.567 us, total = 883.322 ms, Queueing time: mean = 9.260 us, max = 1.017 ms, min = 1.567 us, total = 319.912 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2138 total (0 active), Execution time: mean = 36.806 us, total = 78.691 ms, Queueing time: mean = 84.202 us, max = 1.217 ms, min = 4.084 us, total = 180.023 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2138 total (24 active), Execution time: mean = 93.680 s, total = 200287.760 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1792 total (0 active), Execution time: mean = 102.650 us, total = 183.949 ms, Queueing time: mean = 90.980 us, max = 260.819 us, min = 11.233 us, total = 163.035 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1792 total (0 active), Execution time: mean = 552.666 us, total = 990.377 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 443.052 us, total = 12.849 ms, Queueing time: mean = 68.588 us, max = 114.456 us, min = 33.142 us, total = 1.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:36:15,417 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34595 total (0 active) Queueing time: mean = 9.762 us, max = 2.505 ms, min = 563.000 ns, total = 337.713 ms Execution time: mean = 25.727 us, total = 890.028 ms Event stats: event_loop_lag_probe - 34549 total (0 active), Execution time: mean = 25.655 us, total = 886.344 ms, Queueing time: mean = 9.701 us, max = 2.505 ms, min = 1.873 us, total = 335.157 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:37:15,417 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1736 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1804 - cumulative published bytes: 1433932 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:37:15,418 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 104588 total (4 active) Queueing time: mean = 59.295 us, max = 225.583 ms, min = -0.023 s, total = 6.202 s Execution time: mean = 226.258 us, total = 23.664 s Event stats: event_loop_lag_probe - 34787 total (0 active), Execution time: mean = 25.617 us, total = 891.123 ms, Queueing time: mean = 9.863 us, max = 6.895 ms, min = 1.268 us, total = 343.103 ms GcsInMemoryStore.Put - 13959 total (0 active), Execution time: mean = 21.498 us, total = 300.095 ms, Queueing time: mean = 156.708 us, max = 225.278 ms, min = 1.495 us, total = 2.187 s RayletLoadPulled - 8698 total (1 active), Execution time: mean = 244.596 us, total = 2.127 s, Queueing time: mean = 73.609 us, max = 1.351 ms, min = 10.621 us, total = 640.247 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8696 total (0 active), Execution time: mean = 75.067 us, total = 652.779 ms, Queueing time: mean = 99.925 us, max = 15.090 ms, min = 11.600 us, total = 868.951 ms NodeManagerService.grpc_client.GetResourceLoad - 8696 total (0 active), Execution time: mean = 1.372 ms, total = 11.932 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3487 total (0 active), Execution time: mean = 494.622 us, total = 1.725 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3487 total (0 active), Execution time: mean = 39.774 us, total = 138.692 ms, Queueing time: mean = 90.936 us, max = 601.063 us, min = 9.911 us, total = 317.095 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3478 total (0 active), Execution time: mean = 35.058 us, total = 121.933 ms, Queueing time: mean = 105.677 us, max = 1.496 ms, min = 14.438 us, total = 367.543 ms NodeInfoGcsService.grpc_server.CheckAlive - 3478 total (0 active), Execution time: mean = 514.619 us, total = 1.790 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2900 total (1 active), Execution time: mean = 10.139 us, total = 29.403 ms, Queueing time: mean = 74.243 us, max = 1.648 ms, min = 7.259 us, total = 215.305 ms HealthCheck - 2898 total (0 active), Execution time: mean = 7.634 us, total = 22.124 ms, Queueing time: mean = 91.120 us, max = 1.139 ms, min = 11.335 us, total = 264.066 ms GcsInMemoryStore.Get - 1796 total (0 active), Execution time: mean = 22.395 us, total = 40.221 ms, Queueing time: mean = 12.451 us, max = 137.899 us, min = 2.227 us, total = 22.363 ms InternalKVGcsService.grpc_server.InternalKVGet - 1795 total (0 active), Execution time: mean = 567.251 us, total = 1.018 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1795 total (0 active), Execution time: mean = 55.062 us, total = 98.837 ms, Queueing time: mean = 131.127 us, max = 22.982 ms, min = 9.184 us, total = 235.373 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1736 total (0 active), Execution time: mean = 617.271 us, total = 1.072 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1736 total (0 active), Execution time: mean = 90.806 us, total = 157.639 ms, Queueing time: mean = 105.103 us, max = 273.843 us, min = 15.511 us, total = 182.459 ms GCSServer.deadline_timer.debug_state_dump - 870 total (1 active), Execution time: mean = 1.363 ms, total = 1.186 s, Queueing time: mean = 41.587 us, max = 178.374 us, min = -0.023 s, total = 36.181 ms GCSServer.deadline_timer.debug_state_event_stats_print - 145 total (1 active, 1 running), Execution time: mean = 1.522 ms, total = 220.659 ms, Queueing time: mean = 59.478 us, max = 238.837 us, min = 13.943 us, total = 8.624 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:37:15,418 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 401762 total (1 active) Queueing time: mean = 40.687 us, max = 58.960 ms, min = 1.891 us, total = 16.346 s Execution time: mean = 217.044 us, total = 87.200 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 182616 total (0 active), Execution time: mean = 449.477 us, total = 82.082 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 182616 total (0 active), Execution time: mean = 23.057 us, total = 4.211 s, Queueing time: mean = 87.117 us, max = 58.960 ms, min = 2.005 us, total = 15.909 s event_loop_lag_probe - 34788 total (0 active), Execution time: mean = 25.622 us, total = 891.350 ms, Queueing time: mean = 8.930 us, max = 395.058 us, min = 1.891 us, total = 310.642 ms GcsTaskManager.GcJobSummary - 1741 total (1 active), Execution time: mean = 9.545 us, total = 16.619 ms, Queueing time: mean = 72.778 us, max = 3.642 ms, min = 12.167 us, total = 126.706 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:37:15,418 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 42788 total (25 active) Queueing time: mean = 15.721 us, max = 1.217 ms, min = 1.567 us, total = 672.685 ms Execution time: mean = 4.753 s, total = 203350.108 s Event stats: event_loop_lag_probe - 34789 total (0 active), Execution time: mean = 25.580 us, total = 889.918 ms, Queueing time: mean = 9.264 us, max = 1.017 ms, min = 1.567 us, total = 322.296 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2155 total (0 active), Execution time: mean = 36.868 us, total = 79.451 ms, Queueing time: mean = 84.428 us, max = 1.217 ms, min = 4.084 us, total = 181.943 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2155 total (24 active), Execution time: mean = 94.361 s, total = 203347.906 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1804 total (0 active), Execution time: mean = 102.722 us, total = 185.310 ms, Queueing time: mean = 91.119 us, max = 260.819 us, min = 11.233 us, total = 164.379 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1804 total (0 active), Execution time: mean = 553.447 us, total = 998.419 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 30 total (1 active), Execution time: mean = 432.543 us, total = 12.976 ms, Queueing time: mean = 69.206 us, max = 114.456 us, min = 33.142 us, total = 2.076 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:37:15,418 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34835 total (0 active) Queueing time: mean = 9.762 us, max = 2.505 ms, min = 563.000 ns, total = 340.054 ms Execution time: mean = 25.726 us, total = 896.169 ms Event stats: event_loop_lag_probe - 34789 total (0 active), Execution time: mean = 25.654 us, total = 892.485 ms, Queueing time: mean = 9.701 us, max = 2.505 ms, min = 1.873 us, total = 337.498 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:38:15,418 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1748 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1816 - cumulative published bytes: 1443760 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:38:15,419 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 105307 total (4 active) Queueing time: mean = 59.178 us, max = 225.583 ms, min = -0.023 s, total = 6.232 s Execution time: mean = 226.392 us, total = 23.841 s Event stats: event_loop_lag_probe - 35027 total (0 active), Execution time: mean = 25.613 us, total = 897.152 ms, Queueing time: mean = 9.860 us, max = 6.895 ms, min = 1.268 us, total = 345.371 ms GcsInMemoryStore.Put - 14055 total (0 active), Execution time: mean = 21.395 us, total = 300.711 ms, Queueing time: mean = 155.914 us, max = 225.278 ms, min = 1.495 us, total = 2.191 s RayletLoadPulled - 8758 total (1 active), Execution time: mean = 244.695 us, total = 2.143 s, Queueing time: mean = 73.689 us, max = 1.351 ms, min = 10.621 us, total = 645.365 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8756 total (0 active), Execution time: mean = 75.093 us, total = 657.514 ms, Queueing time: mean = 100.010 us, max = 15.090 ms, min = 11.600 us, total = 875.687 ms NodeManagerService.grpc_client.GetResourceLoad - 8756 total (0 active), Execution time: mean = 1.373 ms, total = 12.026 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3511 total (0 active), Execution time: mean = 495.060 us, total = 1.738 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3511 total (0 active), Execution time: mean = 39.758 us, total = 139.589 ms, Queueing time: mean = 91.048 us, max = 601.063 us, min = 9.911 us, total = 319.668 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3502 total (0 active), Execution time: mean = 35.064 us, total = 122.793 ms, Queueing time: mean = 105.724 us, max = 1.496 ms, min = 14.438 us, total = 370.246 ms NodeInfoGcsService.grpc_server.CheckAlive - 3502 total (0 active), Execution time: mean = 514.901 us, total = 1.803 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2920 total (1 active), Execution time: mean = 10.131 us, total = 29.583 ms, Queueing time: mean = 74.209 us, max = 1.648 ms, min = 7.259 us, total = 216.690 ms HealthCheck - 2918 total (0 active), Execution time: mean = 7.640 us, total = 22.295 ms, Queueing time: mean = 91.234 us, max = 1.139 ms, min = 11.335 us, total = 266.219 ms GcsInMemoryStore.Get - 1808 total (0 active), Execution time: mean = 22.416 us, total = 40.527 ms, Queueing time: mean = 12.462 us, max = 137.899 us, min = 2.227 us, total = 22.532 ms InternalKVGcsService.grpc_server.InternalKVGet - 1807 total (0 active), Execution time: mean = 567.633 us, total = 1.026 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1807 total (0 active), Execution time: mean = 55.047 us, total = 99.469 ms, Queueing time: mean = 131.062 us, max = 22.982 ms, min = 9.184 us, total = 236.829 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1748 total (0 active), Execution time: mean = 617.589 us, total = 1.080 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1748 total (0 active), Execution time: mean = 90.811 us, total = 158.738 ms, Queueing time: mean = 105.175 us, max = 273.843 us, min = 15.511 us, total = 183.845 ms GCSServer.deadline_timer.debug_state_dump - 876 total (1 active), Execution time: mean = 1.363 ms, total = 1.194 s, Queueing time: mean = 41.795 us, max = 178.374 us, min = -0.023 s, total = 36.612 ms GCSServer.deadline_timer.debug_state_event_stats_print - 146 total (1 active, 1 running), Execution time: mean = 1.521 ms, total = 222.112 ms, Queueing time: mean = 59.402 us, max = 238.837 us, min = 13.943 us, total = 8.673 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:38:15,420 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 404534 total (1 active) Queueing time: mean = 40.769 us, max = 58.960 ms, min = 1.891 us, total = 16.492 s Execution time: mean = 217.377 us, total = 87.937 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 183876 total (0 active), Execution time: mean = 450.176 us, total = 82.777 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 183876 total (0 active), Execution time: mean = 23.091 us, total = 4.246 s, Queueing time: mean = 87.298 us, max = 58.960 ms, min = 2.005 us, total = 16.052 s event_loop_lag_probe - 35028 total (0 active), Execution time: mean = 25.617 us, total = 897.303 ms, Queueing time: mean = 8.931 us, max = 395.058 us, min = 1.891 us, total = 312.820 ms GcsTaskManager.GcJobSummary - 1753 total (1 active), Execution time: mean = 9.543 us, total = 16.728 ms, Queueing time: mean = 72.760 us, max = 3.642 ms, min = 12.167 us, total = 127.549 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:38:15,420 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 43076 total (25 active) Queueing time: mean = 15.734 us, max = 1.217 ms, min = 1.567 us, total = 677.778 ms Execution time: mean = 4.722 s, total = 203410.209 s Event stats: event_loop_lag_probe - 35029 total (0 active), Execution time: mean = 25.588 us, total = 896.323 ms, Queueing time: mean = 9.267 us, max = 1.017 ms, min = 1.567 us, total = 324.599 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2167 total (0 active), Execution time: mean = 36.946 us, total = 80.062 ms, Queueing time: mean = 84.585 us, max = 1.217 ms, min = 4.084 us, total = 183.297 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2167 total (24 active), Execution time: mean = 93.866 s, total = 203407.990 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1816 total (0 active), Execution time: mean = 102.717 us, total = 186.535 ms, Queueing time: mean = 91.308 us, max = 260.819 us, min = 11.233 us, total = 165.815 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1816 total (0 active), Execution time: mean = 554.098 us, total = 1.006 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 30 total (1 active), Execution time: mean = 432.543 us, total = 12.976 ms, Queueing time: mean = 69.206 us, max = 114.456 us, min = 33.142 us, total = 2.076 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:38:15,420 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 35075 total (0 active) Queueing time: mean = 9.761 us, max = 2.505 ms, min = 563.000 ns, total = 342.384 ms Execution time: mean = 25.727 us, total = 902.363 ms Event stats: event_loop_lag_probe - 35029 total (0 active), Execution time: mean = 25.655 us, total = 898.679 ms, Queueing time: mean = 9.701 us, max = 2.505 ms, min = 1.873 us, total = 339.828 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:39:15,420 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1760 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1828 - cumulative published bytes: 1453588 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:39:15,421 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 106026 total (4 active) Queueing time: mean = 59.074 us, max = 225.583 ms, min = -0.023 s, total = 6.263 s Execution time: mean = 226.495 us, total = 24.014 s Event stats: event_loop_lag_probe - 35267 total (0 active), Execution time: mean = 25.600 us, total = 902.850 ms, Queueing time: mean = 9.854 us, max = 6.895 ms, min = 1.268 us, total = 347.533 ms GcsInMemoryStore.Put - 14151 total (0 active), Execution time: mean = 21.291 us, total = 301.288 ms, Queueing time: mean = 155.132 us, max = 225.278 ms, min = 1.495 us, total = 2.195 s RayletLoadPulled - 8818 total (1 active), Execution time: mean = 244.754 us, total = 2.158 s, Queueing time: mean = 73.740 us, max = 1.351 ms, min = 10.621 us, total = 650.242 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8816 total (0 active), Execution time: mean = 75.119 us, total = 662.253 ms, Queueing time: mean = 100.068 us, max = 15.090 ms, min = 11.600 us, total = 882.202 ms NodeManagerService.grpc_client.GetResourceLoad - 8816 total (0 active), Execution time: mean = 1.375 ms, total = 12.118 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3535 total (0 active), Execution time: mean = 495.433 us, total = 1.751 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3535 total (0 active), Execution time: mean = 39.730 us, total = 140.444 ms, Queueing time: mean = 91.170 us, max = 601.063 us, min = 9.911 us, total = 322.284 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3526 total (0 active), Execution time: mean = 35.060 us, total = 123.622 ms, Queueing time: mean = 105.778 us, max = 1.496 ms, min = 14.438 us, total = 372.973 ms NodeInfoGcsService.grpc_server.CheckAlive - 3526 total (0 active), Execution time: mean = 515.176 us, total = 1.817 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2940 total (1 active), Execution time: mean = 10.126 us, total = 29.771 ms, Queueing time: mean = 74.771 us, max = 1.648 ms, min = 7.259 us, total = 219.828 ms HealthCheck - 2938 total (0 active), Execution time: mean = 7.644 us, total = 22.459 ms, Queueing time: mean = 91.357 us, max = 1.139 ms, min = 11.335 us, total = 268.406 ms GcsInMemoryStore.Get - 1820 total (0 active), Execution time: mean = 22.433 us, total = 40.827 ms, Queueing time: mean = 12.462 us, max = 137.899 us, min = 2.227 us, total = 22.680 ms InternalKVGcsService.grpc_server.InternalKVGet - 1819 total (0 active), Execution time: mean = 567.871 us, total = 1.033 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1819 total (0 active), Execution time: mean = 55.031 us, total = 100.102 ms, Queueing time: mean = 130.883 us, max = 22.982 ms, min = 9.184 us, total = 238.075 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1760 total (0 active), Execution time: mean = 617.733 us, total = 1.087 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1760 total (0 active), Execution time: mean = 90.815 us, total = 159.835 ms, Queueing time: mean = 105.312 us, max = 273.843 us, min = 15.511 us, total = 185.349 ms GCSServer.deadline_timer.debug_state_dump - 882 total (1 active), Execution time: mean = 1.363 ms, total = 1.202 s, Queueing time: mean = 41.972 us, max = 178.374 us, min = -0.023 s, total = 37.019 ms GCSServer.deadline_timer.debug_state_event_stats_print - 147 total (1 active, 1 running), Execution time: mean = 1.523 ms, total = 223.831 ms, Queueing time: mean = 59.591 us, max = 238.837 us, min = 13.943 us, total = 8.760 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:39:15,421 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 407306 total (1 active) Queueing time: mean = 40.841 us, max = 58.960 ms, min = 1.891 us, total = 16.635 s Execution time: mean = 217.690 us, total = 88.666 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 185136 total (0 active), Execution time: mean = 450.839 us, total = 83.466 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 185136 total (0 active), Execution time: mean = 23.117 us, total = 4.280 s, Queueing time: mean = 87.457 us, max = 58.960 ms, min = 2.005 us, total = 16.191 s event_loop_lag_probe - 35268 total (0 active), Execution time: mean = 25.611 us, total = 903.242 ms, Queueing time: mean = 8.931 us, max = 395.058 us, min = 1.891 us, total = 314.963 ms GcsTaskManager.GcJobSummary - 1765 total (1 active), Execution time: mean = 9.544 us, total = 16.844 ms, Queueing time: mean = 72.742 us, max = 3.642 ms, min = 12.167 us, total = 128.389 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:39:15,422 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 43364 total (25 active) Queueing time: mean = 15.743 us, max = 1.217 ms, min = 1.567 us, total = 682.674 ms Execution time: mean = 4.692 s, total = 203470.436 s Event stats: event_loop_lag_probe - 35269 total (0 active), Execution time: mean = 25.583 us, total = 902.287 ms, Queueing time: mean = 9.265 us, max = 1.017 ms, min = 1.567 us, total = 326.750 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2179 total (0 active), Execution time: mean = 36.956 us, total = 80.528 ms, Queueing time: mean = 84.736 us, max = 1.217 ms, min = 4.084 us, total = 184.640 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2179 total (24 active), Execution time: mean = 93.377 s, total = 203468.203 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1828 total (0 active), Execution time: mean = 102.655 us, total = 187.653 ms, Queueing time: mean = 91.475 us, max = 260.819 us, min = 11.233 us, total = 167.217 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1828 total (0 active), Execution time: mean = 554.640 us, total = 1.014 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 30 total (1 active), Execution time: mean = 432.543 us, total = 12.976 ms, Queueing time: mean = 69.206 us, max = 114.456 us, min = 33.142 us, total = 2.076 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:39:15,422 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 35315 total (0 active) Queueing time: mean = 9.758 us, max = 2.505 ms, min = 563.000 ns, total = 344.619 ms Execution time: mean = 25.719 us, total = 908.271 ms Event stats: event_loop_lag_probe - 35269 total (0 active), Execution time: mean = 25.648 us, total = 904.586 ms, Queueing time: mean = 9.699 us, max = 2.505 ms, min = 1.873 us, total = 342.063 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:40:15,422 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1772 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1840 - cumulative published bytes: 1463416 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:40:15,423 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 106745 total (4 active) Queueing time: mean = 58.956 us, max = 225.583 ms, min = -0.023 s, total = 6.293 s Execution time: mean = 226.615 us, total = 24.190 s Event stats: event_loop_lag_probe - 35507 total (0 active), Execution time: mean = 25.598 us, total = 908.899 ms, Queueing time: mean = 9.852 us, max = 6.895 ms, min = 1.268 us, total = 349.807 ms GcsInMemoryStore.Put - 14247 total (0 active), Execution time: mean = 21.188 us, total = 301.868 ms, Queueing time: mean = 154.360 us, max = 225.278 ms, min = 1.495 us, total = 2.199 s RayletLoadPulled - 8878 total (1 active), Execution time: mean = 244.826 us, total = 2.174 s, Queueing time: mean = 73.780 us, max = 1.351 ms, min = 10.621 us, total = 655.022 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8876 total (0 active), Execution time: mean = 75.143 us, total = 666.971 ms, Queueing time: mean = 100.162 us, max = 15.090 ms, min = 11.600 us, total = 889.038 ms NodeManagerService.grpc_client.GetResourceLoad - 8876 total (0 active), Execution time: mean = 1.376 ms, total = 12.211 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3559 total (0 active), Execution time: mean = 495.880 us, total = 1.765 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3559 total (0 active), Execution time: mean = 39.693 us, total = 141.266 ms, Queueing time: mean = 91.305 us, max = 601.063 us, min = 9.911 us, total = 324.955 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3550 total (0 active), Execution time: mean = 35.064 us, total = 124.477 ms, Queueing time: mean = 105.868 us, max = 1.496 ms, min = 14.438 us, total = 375.831 ms NodeInfoGcsService.grpc_server.CheckAlive - 3550 total (0 active), Execution time: mean = 515.560 us, total = 1.830 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2960 total (1 active), Execution time: mean = 10.121 us, total = 29.958 ms, Queueing time: mean = 74.739 us, max = 1.648 ms, min = 7.259 us, total = 221.228 ms HealthCheck - 2958 total (0 active), Execution time: mean = 7.650 us, total = 22.630 ms, Queueing time: mean = 91.386 us, max = 1.139 ms, min = 11.335 us, total = 270.319 ms GcsInMemoryStore.Get - 1832 total (0 active), Execution time: mean = 22.451 us, total = 41.130 ms, Queueing time: mean = 12.464 us, max = 137.899 us, min = 2.227 us, total = 22.834 ms InternalKVGcsService.grpc_server.InternalKVGet - 1831 total (0 active), Execution time: mean = 568.057 us, total = 1.040 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1831 total (0 active), Execution time: mean = 55.017 us, total = 100.735 ms, Queueing time: mean = 130.729 us, max = 22.982 ms, min = 9.184 us, total = 239.365 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1772 total (0 active), Execution time: mean = 617.769 us, total = 1.095 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1772 total (0 active), Execution time: mean = 90.758 us, total = 160.823 ms, Queueing time: mean = 105.353 us, max = 273.843 us, min = 15.511 us, total = 186.686 ms GCSServer.deadline_timer.debug_state_dump - 888 total (1 active), Execution time: mean = 1.363 ms, total = 1.211 s, Queueing time: mean = 42.181 us, max = 178.374 us, min = -0.023 s, total = 37.456 ms GCSServer.deadline_timer.debug_state_event_stats_print - 148 total (1 active, 1 running), Execution time: mean = 1.524 ms, total = 225.478 ms, Queueing time: mean = 59.635 us, max = 238.837 us, min = 13.943 us, total = 8.826 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:40:15,423 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 410078 total (1 active) Queueing time: mean = 40.921 us, max = 58.960 ms, min = 1.891 us, total = 16.781 s Execution time: mean = 217.994 us, total = 89.395 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 186396 total (0 active), Execution time: mean = 451.480 us, total = 84.154 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 186396 total (0 active), Execution time: mean = 23.146 us, total = 4.314 s, Queueing time: mean = 87.632 us, max = 58.960 ms, min = 2.005 us, total = 16.334 s event_loop_lag_probe - 35508 total (0 active), Execution time: mean = 25.608 us, total = 909.292 ms, Queueing time: mean = 8.932 us, max = 395.058 us, min = 1.891 us, total = 317.164 ms GcsTaskManager.GcJobSummary - 1777 total (1 active), Execution time: mean = 9.537 us, total = 16.948 ms, Queueing time: mean = 72.828 us, max = 3.642 ms, min = 12.167 us, total = 129.415 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:40:15,423 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 43651 total (25 active) Queueing time: mean = 15.745 us, max = 1.217 ms, min = 1.567 us, total = 687.300 ms Execution time: mean = 4.663 s, total = 203530.615 s Event stats: event_loop_lag_probe - 35508 total (0 active), Execution time: mean = 25.591 us, total = 908.675 ms, Queueing time: mean = 9.267 us, max = 1.017 ms, min = 1.567 us, total = 329.056 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2191 total (0 active), Execution time: mean = 37.019 us, total = 81.108 ms, Queueing time: mean = 84.817 us, max = 1.217 ms, min = 4.084 us, total = 185.835 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2191 total (24 active), Execution time: mean = 92.893 s, total = 203528.366 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1840 total (0 active), Execution time: mean = 102.787 us, total = 189.129 ms, Queueing time: mean = 91.490 us, max = 260.819 us, min = 11.233 us, total = 168.341 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1840 total (0 active), Execution time: mean = 555.010 us, total = 1.021 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 30 total (1 active), Execution time: mean = 432.543 us, total = 12.976 ms, Queueing time: mean = 69.206 us, max = 114.456 us, min = 33.142 us, total = 2.076 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:40:15,423 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 35555 total (0 active) Queueing time: mean = 9.760 us, max = 2.505 ms, min = 563.000 ns, total = 347.016 ms Execution time: mean = 25.717 us, total = 914.361 ms Event stats: event_loop_lag_probe - 35509 total (0 active), Execution time: mean = 25.646 us, total = 910.677 ms, Queueing time: mean = 9.701 us, max = 2.505 ms, min = 1.873 us, total = 344.460 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:41:15,424 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1784 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1852 - cumulative published bytes: 1473244 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:41:15,425 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 107464 total (4 active) Queueing time: mean = 58.778 us, max = 225.583 ms, min = -0.023 s, total = 6.317 s Execution time: mean = 226.541 us, total = 24.345 s Event stats: event_loop_lag_probe - 35747 total (0 active), Execution time: mean = 25.586 us, total = 914.607 ms, Queueing time: mean = 9.846 us, max = 6.895 ms, min = 1.268 us, total = 351.950 ms GcsInMemoryStore.Put - 14343 total (0 active), Execution time: mean = 21.080 us, total = 302.354 ms, Queueing time: mean = 153.592 us, max = 225.278 ms, min = 1.495 us, total = 2.203 s RayletLoadPulled - 8938 total (1 active), Execution time: mean = 244.799 us, total = 2.188 s, Queueing time: mean = 73.736 us, max = 1.351 ms, min = 10.621 us, total = 659.053 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8936 total (0 active), Execution time: mean = 75.112 us, total = 671.204 ms, Queueing time: mean = 100.017 us, max = 15.090 ms, min = 11.188 us, total = 893.755 ms NodeManagerService.grpc_client.GetResourceLoad - 8936 total (0 active), Execution time: mean = 1.376 ms, total = 12.294 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3583 total (0 active), Execution time: mean = 495.405 us, total = 1.775 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3583 total (0 active), Execution time: mean = 39.646 us, total = 142.053 ms, Queueing time: mean = 91.197 us, max = 601.063 us, min = 9.911 us, total = 326.761 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3574 total (0 active), Execution time: mean = 35.046 us, total = 125.255 ms, Queueing time: mean = 105.703 us, max = 1.496 ms, min = 13.520 us, total = 377.784 ms NodeInfoGcsService.grpc_server.CheckAlive - 3574 total (0 active), Execution time: mean = 515.271 us, total = 1.842 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2980 total (1 active), Execution time: mean = 10.111 us, total = 30.132 ms, Queueing time: mean = 74.645 us, max = 1.648 ms, min = 7.259 us, total = 222.442 ms HealthCheck - 2978 total (0 active), Execution time: mean = 7.645 us, total = 22.767 ms, Queueing time: mean = 91.176 us, max = 1.139 ms, min = 11.335 us, total = 271.521 ms GcsInMemoryStore.Get - 1844 total (0 active), Execution time: mean = 22.441 us, total = 41.382 ms, Queueing time: mean = 12.450 us, max = 137.899 us, min = 2.227 us, total = 22.958 ms InternalKVGcsService.grpc_server.InternalKVGet - 1843 total (0 active), Execution time: mean = 567.313 us, total = 1.046 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1843 total (0 active), Execution time: mean = 54.992 us, total = 101.350 ms, Queueing time: mean = 130.326 us, max = 22.982 ms, min = 9.184 us, total = 240.191 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1784 total (0 active), Execution time: mean = 617.228 us, total = 1.101 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1784 total (0 active), Execution time: mean = 90.657 us, total = 161.732 ms, Queueing time: mean = 105.212 us, max = 273.843 us, min = 15.511 us, total = 187.698 ms GCSServer.deadline_timer.debug_state_dump - 894 total (1 active), Execution time: mean = 1.364 ms, total = 1.219 s, Queueing time: mean = 42.295 us, max = 178.374 us, min = -0.023 s, total = 37.812 ms GCSServer.deadline_timer.debug_state_event_stats_print - 149 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 227.165 ms, Queueing time: mean = 59.709 us, max = 238.837 us, min = 13.943 us, total = 8.897 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:41:15,425 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 412850 total (1 active) Queueing time: mean = 40.888 us, max = 58.960 ms, min = 1.891 us, total = 16.881 s Execution time: mean = 217.885 us, total = 89.954 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 187656 total (0 active), Execution time: mean = 451.248 us, total = 84.679 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 187656 total (0 active), Execution time: mean = 23.138 us, total = 4.342 s, Queueing time: mean = 87.560 us, max = 58.960 ms, min = 2.005 us, total = 16.431 s event_loop_lag_probe - 35748 total (0 active), Execution time: mean = 25.605 us, total = 915.323 ms, Queueing time: mean = 8.931 us, max = 395.058 us, min = 1.891 us, total = 319.249 ms GcsTaskManager.GcJobSummary - 1789 total (1 active), Execution time: mean = 9.535 us, total = 17.057 ms, Queueing time: mean = 72.778 us, max = 3.642 ms, min = 12.167 us, total = 130.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:41:15,425 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 43939 total (25 active) Queueing time: mean = 15.716 us, max = 1.217 ms, min = 1.567 us, total = 690.531 ms Execution time: mean = 4.633 s, total = 203590.789 s Event stats: event_loop_lag_probe - 35748 total (0 active), Execution time: mean = 25.574 us, total = 914.224 ms, Queueing time: mean = 9.260 us, max = 1.017 ms, min = 1.567 us, total = 331.031 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2203 total (0 active), Execution time: mean = 36.986 us, total = 81.480 ms, Queueing time: mean = 84.663 us, max = 1.217 ms, min = 4.084 us, total = 186.513 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2203 total (24 active), Execution time: mean = 92.414 s, total = 203588.528 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1852 total (0 active), Execution time: mean = 102.649 us, total = 190.106 ms, Queueing time: mean = 91.209 us, max = 260.819 us, min = 11.233 us, total = 168.920 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1852 total (0 active), Execution time: mean = 554.088 us, total = 1.026 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 30 total (1 active), Execution time: mean = 432.543 us, total = 12.976 ms, Queueing time: mean = 69.206 us, max = 114.456 us, min = 33.142 us, total = 2.076 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:41:15,425 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 35795 total (0 active) Queueing time: mean = 9.758 us, max = 2.505 ms, min = 563.000 ns, total = 349.302 ms Execution time: mean = 25.707 us, total = 920.191 ms Event stats: event_loop_lag_probe - 35749 total (0 active), Execution time: mean = 25.637 us, total = 916.507 ms, Queueing time: mean = 9.699 us, max = 2.505 ms, min = 1.873 us, total = 346.746 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:42:15,425 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1796 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1865 - cumulative published bytes: 1483106 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:42:15,426 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 108183 total (4 active) Queueing time: mean = 58.592 us, max = 225.583 ms, min = -0.023 s, total = 6.339 s Execution time: mean = 226.408 us, total = 24.493 s Event stats: event_loop_lag_probe - 35987 total (0 active), Execution time: mean = 25.572 us, total = 920.274 ms, Queueing time: mean = 9.841 us, max = 6.895 ms, min = 1.268 us, total = 354.142 ms GcsInMemoryStore.Put - 14439 total (0 active), Execution time: mean = 20.972 us, total = 302.819 ms, Queueing time: mean = 152.826 us, max = 225.278 ms, min = 1.495 us, total = 2.207 s RayletLoadPulled - 8998 total (1 active), Execution time: mean = 244.801 us, total = 2.203 s, Queueing time: mean = 73.677 us, max = 1.351 ms, min = 10.621 us, total = 662.945 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8996 total (0 active), Execution time: mean = 75.117 us, total = 675.749 ms, Queueing time: mean = 99.851 us, max = 15.090 ms, min = 11.188 us, total = 898.259 ms NodeManagerService.grpc_client.GetResourceLoad - 8996 total (0 active), Execution time: mean = 1.375 ms, total = 12.370 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3607 total (0 active), Execution time: mean = 494.688 us, total = 1.784 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3607 total (0 active), Execution time: mean = 39.609 us, total = 142.869 ms, Queueing time: mean = 90.931 us, max = 601.063 us, min = 9.911 us, total = 327.987 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3598 total (0 active), Execution time: mean = 35.034 us, total = 126.053 ms, Queueing time: mean = 105.535 us, max = 1.496 ms, min = 13.520 us, total = 379.716 ms NodeInfoGcsService.grpc_server.CheckAlive - 3598 total (0 active), Execution time: mean = 514.985 us, total = 1.853 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3000 total (1 active), Execution time: mean = 10.105 us, total = 30.315 ms, Queueing time: mean = 74.574 us, max = 1.648 ms, min = 7.259 us, total = 223.722 ms HealthCheck - 2998 total (0 active), Execution time: mean = 7.640 us, total = 22.904 ms, Queueing time: mean = 91.024 us, max = 1.139 ms, min = 11.335 us, total = 272.889 ms GcsInMemoryStore.Get - 1856 total (0 active), Execution time: mean = 22.436 us, total = 41.640 ms, Queueing time: mean = 12.437 us, max = 137.899 us, min = 2.227 us, total = 23.082 ms InternalKVGcsService.grpc_server.InternalKVGet - 1855 total (0 active), Execution time: mean = 566.496 us, total = 1.051 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1855 total (0 active), Execution time: mean = 54.963 us, total = 101.957 ms, Queueing time: mean = 129.827 us, max = 22.982 ms, min = 9.184 us, total = 240.829 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1796 total (0 active), Execution time: mean = 616.581 us, total = 1.107 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1796 total (0 active), Execution time: mean = 90.633 us, total = 162.776 ms, Queueing time: mean = 104.948 us, max = 273.843 us, min = 15.511 us, total = 188.487 ms GCSServer.deadline_timer.debug_state_dump - 900 total (1 active), Execution time: mean = 1.365 ms, total = 1.228 s, Queueing time: mean = 42.507 us, max = 178.374 us, min = -0.023 s, total = 38.256 ms GCSServer.deadline_timer.debug_state_event_stats_print - 150 total (1 active, 1 running), Execution time: mean = 1.525 ms, total = 228.740 ms, Queueing time: mean = 59.667 us, max = 238.837 us, min = 13.943 us, total = 8.950 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:42:15,426 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 415622 total (1 active) Queueing time: mean = 40.818 us, max = 58.960 ms, min = 1.891 us, total = 16.965 s Execution time: mean = 217.699 us, total = 90.481 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 188916 total (0 active), Execution time: mean = 450.862 us, total = 85.175 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 188916 total (0 active), Execution time: mean = 23.118 us, total = 4.367 s, Queueing time: mean = 87.407 us, max = 58.960 ms, min = 2.005 us, total = 16.513 s event_loop_lag_probe - 35988 total (0 active), Execution time: mean = 25.598 us, total = 921.237 ms, Queueing time: mean = 8.929 us, max = 395.058 us, min = 1.891 us, total = 321.346 ms GcsTaskManager.GcJobSummary - 1801 total (1 active), Execution time: mean = 9.527 us, total = 17.157 ms, Queueing time: mean = 72.680 us, max = 3.642 ms, min = 12.167 us, total = 130.897 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:42:15,427 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 44268 total (25 active) Queueing time: mean = 15.709 us, max = 1.217 ms, min = 1.567 us, total = 695.406 ms Execution time: mean = 4.844 s, total = 214450.945 s Event stats: event_loop_lag_probe - 35988 total (0 active), Execution time: mean = 25.572 us, total = 920.294 ms, Queueing time: mean = 9.259 us, max = 1.017 ms, min = 1.567 us, total = 333.216 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2234 total (0 active), Execution time: mean = 36.825 us, total = 82.267 ms, Queueing time: mean = 84.111 us, max = 1.217 ms, min = 4.084 us, total = 187.905 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2234 total (24 active), Execution time: mean = 95.993 s, total = 214448.668 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1865 total (0 active), Execution time: mean = 102.619 us, total = 191.385 ms, Queueing time: mean = 91.241 us, max = 260.819 us, min = 11.233 us, total = 170.165 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1865 total (0 active), Execution time: mean = 553.874 us, total = 1.033 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 31 total (1 active), Execution time: mean = 441.088 us, total = 13.674 ms, Queueing time: mean = 68.681 us, max = 114.456 us, min = 33.142 us, total = 2.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:42:15,427 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 36035 total (0 active) Queueing time: mean = 9.757 us, max = 2.505 ms, min = 563.000 ns, total = 351.593 ms Execution time: mean = 25.700 us, total = 926.095 ms Event stats: event_loop_lag_probe - 35989 total (0 active), Execution time: mean = 25.630 us, total = 922.411 ms, Queueing time: mean = 9.698 us, max = 2.505 ms, min = 1.873 us, total = 349.036 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:43:15,427 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1808 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1877 - cumulative published bytes: 1492934 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:43:15,428 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 108902 total (4 active) Queueing time: mean = 58.441 us, max = 225.583 ms, min = -0.023 s, total = 6.364 s Execution time: mean = 226.369 us, total = 24.652 s Event stats: event_loop_lag_probe - 36227 total (0 active), Execution time: mean = 25.550 us, total = 925.613 ms, Queueing time: mean = 9.832 us, max = 6.895 ms, min = 1.268 us, total = 356.178 ms GcsInMemoryStore.Put - 14535 total (0 active), Execution time: mean = 20.874 us, total = 303.410 ms, Queueing time: mean = 152.081 us, max = 225.278 ms, min = 1.495 us, total = 2.210 s RayletLoadPulled - 9058 total (1 active), Execution time: mean = 244.784 us, total = 2.217 s, Queueing time: mean = 73.634 us, max = 1.351 ms, min = 10.621 us, total = 666.980 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9056 total (0 active), Execution time: mean = 75.087 us, total = 679.986 ms, Queueing time: mean = 99.797 us, max = 15.090 ms, min = 11.188 us, total = 903.763 ms NodeManagerService.grpc_client.GetResourceLoad - 9056 total (0 active), Execution time: mean = 1.375 ms, total = 12.453 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3631 total (0 active), Execution time: mean = 494.743 us, total = 1.796 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3631 total (0 active), Execution time: mean = 39.611 us, total = 143.828 ms, Queueing time: mean = 90.923 us, max = 601.063 us, min = 9.911 us, total = 330.140 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3622 total (0 active), Execution time: mean = 35.033 us, total = 126.889 ms, Queueing time: mean = 105.405 us, max = 1.496 ms, min = 13.520 us, total = 381.778 ms NodeInfoGcsService.grpc_server.CheckAlive - 3622 total (0 active), Execution time: mean = 514.827 us, total = 1.865 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3020 total (1 active), Execution time: mean = 10.093 us, total = 30.481 ms, Queueing time: mean = 74.526 us, max = 1.648 ms, min = 7.259 us, total = 225.068 ms HealthCheck - 3018 total (0 active), Execution time: mean = 7.639 us, total = 23.053 ms, Queueing time: mean = 91.008 us, max = 1.139 ms, min = 11.335 us, total = 274.661 ms GcsInMemoryStore.Get - 1868 total (0 active), Execution time: mean = 22.454 us, total = 41.944 ms, Queueing time: mean = 12.432 us, max = 137.899 us, min = 2.227 us, total = 23.224 ms InternalKVGcsService.grpc_server.InternalKVGet - 1867 total (0 active), Execution time: mean = 566.293 us, total = 1.057 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1867 total (0 active), Execution time: mean = 54.911 us, total = 102.520 ms, Queueing time: mean = 129.583 us, max = 22.982 ms, min = 9.184 us, total = 241.932 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1808 total (0 active), Execution time: mean = 616.358 us, total = 1.114 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1808 total (0 active), Execution time: mean = 90.629 us, total = 163.858 ms, Queueing time: mean = 104.988 us, max = 273.843 us, min = 15.511 us, total = 189.819 ms GCSServer.deadline_timer.debug_state_dump - 906 total (1 active), Execution time: mean = 1.365 ms, total = 1.236 s, Queueing time: mean = 42.570 us, max = 178.374 us, min = -0.023 s, total = 38.568 ms GCSServer.deadline_timer.debug_state_event_stats_print - 151 total (1 active, 1 running), Execution time: mean = 1.526 ms, total = 230.391 ms, Queueing time: mean = 59.508 us, max = 238.837 us, min = 13.943 us, total = 8.986 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:43:15,428 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 418393 total (1 active) Queueing time: mean = 40.820 us, max = 58.960 ms, min = 1.891 us, total = 17.079 s Execution time: mean = 217.725 us, total = 91.094 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 190176 total (0 active), Execution time: mean = 450.918 us, total = 85.754 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 190176 total (0 active), Execution time: mean = 23.118 us, total = 4.397 s, Queueing time: mean = 87.412 us, max = 58.960 ms, min = 2.005 us, total = 16.624 s event_loop_lag_probe - 36227 total (0 active), Execution time: mean = 25.586 us, total = 926.897 ms, Queueing time: mean = 8.926 us, max = 395.058 us, min = 1.891 us, total = 323.377 ms GcsTaskManager.GcJobSummary - 1813 total (1 active), Execution time: mean = 9.515 us, total = 17.252 ms, Queueing time: mean = 72.658 us, max = 3.642 ms, min = 12.167 us, total = 131.729 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:43:15,428 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 44556 total (25 active) Queueing time: mean = 15.705 us, max = 1.217 ms, min = 1.567 us, total = 699.765 ms Execution time: mean = 4.814 s, total = 214511.046 s Event stats: event_loop_lag_probe - 36228 total (0 active), Execution time: mean = 25.561 us, total = 926.035 ms, Queueing time: mean = 9.255 us, max = 1.017 ms, min = 1.567 us, total = 335.283 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2246 total (0 active), Execution time: mean = 36.875 us, total = 82.820 ms, Queueing time: mean = 84.178 us, max = 1.217 ms, min = 4.084 us, total = 189.065 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2246 total (24 active), Execution time: mean = 95.507 s, total = 214508.754 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1877 total (0 active), Execution time: mean = 102.626 us, total = 192.628 ms, Queueing time: mean = 91.261 us, max = 260.819 us, min = 11.233 us, total = 171.297 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1877 total (0 active), Execution time: mean = 553.996 us, total = 1.040 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 31 total (1 active), Execution time: mean = 441.088 us, total = 13.674 ms, Queueing time: mean = 68.681 us, max = 114.456 us, min = 33.142 us, total = 2.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:43:15,428 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 36275 total (0 active) Queueing time: mean = 9.752 us, max = 2.505 ms, min = 563.000 ns, total = 353.745 ms Execution time: mean = 25.689 us, total = 931.856 ms Event stats: event_loop_lag_probe - 36229 total (0 active), Execution time: mean = 25.620 us, total = 928.172 ms, Queueing time: mean = 9.694 us, max = 2.505 ms, min = 1.873 us, total = 351.189 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:44:15,429 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1819 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1888 - cumulative published bytes: 1501943 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:44:15,430 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 109609 total (4 active) Queueing time: mean = 58.333 us, max = 225.583 ms, min = -0.023 s, total = 6.394 s Execution time: mean = 226.498 us, total = 24.826 s Event stats: event_loop_lag_probe - 36467 total (0 active), Execution time: mean = 25.544 us, total = 931.501 ms, Queueing time: mean = 9.827 us, max = 6.895 ms, min = 1.268 us, total = 358.361 ms GcsInMemoryStore.Put - 14629 total (0 active), Execution time: mean = 20.780 us, total = 303.985 ms, Queueing time: mean = 151.397 us, max = 225.278 ms, min = 1.495 us, total = 2.215 s RayletLoadPulled - 9118 total (1 active), Execution time: mean = 244.852 us, total = 2.233 s, Queueing time: mean = 73.644 us, max = 1.351 ms, min = 10.621 us, total = 671.488 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9116 total (0 active), Execution time: mean = 75.123 us, total = 684.819 ms, Queueing time: mean = 99.889 us, max = 15.090 ms, min = 11.188 us, total = 910.586 ms NodeManagerService.grpc_client.GetResourceLoad - 9116 total (0 active), Execution time: mean = 1.376 ms, total = 12.547 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3653 total (0 active), Execution time: mean = 495.075 us, total = 1.809 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3653 total (0 active), Execution time: mean = 39.609 us, total = 144.690 ms, Queueing time: mean = 91.005 us, max = 601.063 us, min = 9.911 us, total = 332.440 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3646 total (0 active), Execution time: mean = 35.072 us, total = 127.871 ms, Queueing time: mean = 105.521 us, max = 1.496 ms, min = 13.520 us, total = 384.729 ms NodeInfoGcsService.grpc_server.CheckAlive - 3646 total (0 active), Execution time: mean = 515.256 us, total = 1.879 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3040 total (1 active), Execution time: mean = 10.086 us, total = 30.660 ms, Queueing time: mean = 74.525 us, max = 1.648 ms, min = 7.259 us, total = 226.556 ms HealthCheck - 3037 total (0 active), Execution time: mean = 7.647 us, total = 23.225 ms, Queueing time: mean = 91.135 us, max = 1.139 ms, min = 11.335 us, total = 276.777 ms GcsInMemoryStore.Get - 1879 total (0 active), Execution time: mean = 22.483 us, total = 42.246 ms, Queueing time: mean = 12.439 us, max = 137.899 us, min = 2.227 us, total = 23.372 ms InternalKVGcsService.grpc_server.InternalKVGet - 1878 total (0 active), Execution time: mean = 566.451 us, total = 1.064 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1878 total (0 active), Execution time: mean = 54.880 us, total = 103.065 ms, Queueing time: mean = 129.453 us, max = 22.982 ms, min = 9.184 us, total = 243.112 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1819 total (0 active), Execution time: mean = 616.526 us, total = 1.121 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1819 total (0 active), Execution time: mean = 90.688 us, total = 164.961 ms, Queueing time: mean = 104.906 us, max = 273.843 us, min = 15.511 us, total = 190.824 ms GCSServer.deadline_timer.debug_state_dump - 912 total (1 active), Execution time: mean = 1.365 ms, total = 1.245 s, Queueing time: mean = 42.785 us, max = 178.374 us, min = -0.023 s, total = 39.020 ms GCSServer.deadline_timer.debug_state_event_stats_print - 152 total (1 active, 1 running), Execution time: mean = 1.526 ms, total = 231.892 ms, Queueing time: mean = 59.419 us, max = 238.837 us, min = 13.943 us, total = 9.032 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:44:15,430 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 421165 total (1 active) Queueing time: mean = 40.882 us, max = 58.960 ms, min = 1.891 us, total = 17.218 s Execution time: mean = 218.004 us, total = 91.816 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 191436 total (0 active), Execution time: mean = 451.510 us, total = 86.435 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 191436 total (0 active), Execution time: mean = 23.143 us, total = 4.430 s, Queueing time: mean = 87.547 us, max = 58.960 ms, min = 2.005 us, total = 16.760 s event_loop_lag_probe - 36467 total (0 active), Execution time: mean = 25.578 us, total = 932.758 ms, Queueing time: mean = 8.926 us, max = 395.058 us, min = 1.891 us, total = 325.497 ms GcsTaskManager.GcJobSummary - 1825 total (1 active), Execution time: mean = 9.508 us, total = 17.352 ms, Queueing time: mean = 72.763 us, max = 3.642 ms, min = 12.167 us, total = 132.793 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:44:15,430 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 44840 total (25 active) Queueing time: mean = 15.709 us, max = 1.217 ms, min = 1.567 us, total = 704.370 ms Execution time: mean = 4.785 s, total = 214566.204 s Event stats: event_loop_lag_probe - 36468 total (0 active), Execution time: mean = 25.564 us, total = 932.256 ms, Queueing time: mean = 9.255 us, max = 1.017 ms, min = 1.567 us, total = 337.493 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2257 total (0 active), Execution time: mean = 36.936 us, total = 83.364 ms, Queueing time: mean = 84.267 us, max = 1.217 ms, min = 4.084 us, total = 190.191 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2257 total (24 active), Execution time: mean = 95.066 s, total = 214563.897 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1888 total (0 active), Execution time: mean = 102.633 us, total = 193.771 ms, Queueing time: mean = 91.401 us, max = 260.819 us, min = 11.233 us, total = 172.565 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1888 total (0 active), Execution time: mean = 554.621 us, total = 1.047 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 31 total (1 active), Execution time: mean = 441.088 us, total = 13.674 ms, Queueing time: mean = 68.681 us, max = 114.456 us, min = 33.142 us, total = 2.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:44:15,430 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 36514 total (0 active) Queueing time: mean = 9.751 us, max = 2.505 ms, min = 563.000 ns, total = 356.038 ms Execution time: mean = 25.685 us, total = 937.865 ms Event stats: event_loop_lag_probe - 36468 total (0 active), Execution time: mean = 25.616 us, total = 934.181 ms, Queueing time: mean = 9.693 us, max = 2.505 ms, min = 1.873 us, total = 353.482 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:45:15,430 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1831 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1900 - cumulative published bytes: 1511771 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:45:15,431 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 110328 total (4 active) Queueing time: mean = 58.228 us, max = 225.583 ms, min = -0.023 s, total = 6.424 s Execution time: mean = 226.605 us, total = 25.001 s Event stats: event_loop_lag_probe - 36707 total (0 active), Execution time: mean = 25.532 us, total = 937.216 ms, Queueing time: mean = 9.821 us, max = 6.895 ms, min = 1.268 us, total = 360.516 ms GcsInMemoryStore.Put - 14725 total (0 active), Execution time: mean = 20.687 us, total = 304.609 ms, Queueing time: mean = 150.700 us, max = 225.278 ms, min = 1.495 us, total = 2.219 s RayletLoadPulled - 9178 total (1 active), Execution time: mean = 244.884 us, total = 2.248 s, Queueing time: mean = 73.672 us, max = 1.351 ms, min = 10.621 us, total = 676.160 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9176 total (0 active), Execution time: mean = 75.139 us, total = 689.478 ms, Queueing time: mean = 99.987 us, max = 15.090 ms, min = 11.188 us, total = 917.481 ms NodeManagerService.grpc_client.GetResourceLoad - 9176 total (0 active), Execution time: mean = 1.377 ms, total = 12.639 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3677 total (0 active), Execution time: mean = 495.546 us, total = 1.822 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3677 total (0 active), Execution time: mean = 39.601 us, total = 145.611 ms, Queueing time: mean = 91.127 us, max = 601.063 us, min = 9.911 us, total = 335.074 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3670 total (0 active), Execution time: mean = 35.078 us, total = 128.736 ms, Queueing time: mean = 105.521 us, max = 1.496 ms, min = 13.520 us, total = 387.262 ms NodeInfoGcsService.grpc_server.CheckAlive - 3670 total (0 active), Execution time: mean = 515.547 us, total = 1.892 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3060 total (1 active), Execution time: mean = 10.088 us, total = 30.869 ms, Queueing time: mean = 74.526 us, max = 1.648 ms, min = 7.259 us, total = 228.050 ms HealthCheck - 3057 total (0 active), Execution time: mean = 7.648 us, total = 23.379 ms, Queueing time: mean = 91.260 us, max = 1.139 ms, min = 11.335 us, total = 278.982 ms GcsInMemoryStore.Get - 1891 total (0 active), Execution time: mean = 22.498 us, total = 42.544 ms, Queueing time: mean = 12.447 us, max = 137.899 us, min = 2.227 us, total = 23.538 ms InternalKVGcsService.grpc_server.InternalKVGet - 1890 total (0 active), Execution time: mean = 566.673 us, total = 1.071 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1890 total (0 active), Execution time: mean = 54.865 us, total = 103.694 ms, Queueing time: mean = 129.361 us, max = 22.982 ms, min = 9.184 us, total = 244.493 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1831 total (0 active), Execution time: mean = 616.742 us, total = 1.129 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1831 total (0 active), Execution time: mean = 90.729 us, total = 166.125 ms, Queueing time: mean = 104.926 us, max = 273.843 us, min = 15.511 us, total = 192.120 ms GCSServer.deadline_timer.debug_state_dump - 918 total (1 active), Execution time: mean = 1.365 ms, total = 1.253 s, Queueing time: mean = 43.121 us, max = 214.701 us, min = -0.023 s, total = 39.585 ms GCSServer.deadline_timer.debug_state_event_stats_print - 153 total (1 active, 1 running), Execution time: mean = 1.527 ms, total = 233.615 ms, Queueing time: mean = 59.518 us, max = 238.837 us, min = 13.943 us, total = 9.106 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:45:15,432 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 423937 total (1 active) Queueing time: mean = 40.943 us, max = 58.960 ms, min = 1.891 us, total = 17.357 s Execution time: mean = 218.253 us, total = 92.526 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 192696 total (0 active), Execution time: mean = 452.038 us, total = 87.106 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 192696 total (0 active), Execution time: mean = 23.164 us, total = 4.464 s, Queueing time: mean = 87.684 us, max = 58.960 ms, min = 2.005 us, total = 16.896 s event_loop_lag_probe - 36707 total (0 active), Execution time: mean = 25.571 us, total = 938.621 ms, Queueing time: mean = 8.925 us, max = 395.058 us, min = 1.891 us, total = 327.608 ms GcsTaskManager.GcJobSummary - 1837 total (1 active), Execution time: mean = 9.496 us, total = 17.445 ms, Queueing time: mean = 72.691 us, max = 3.642 ms, min = 12.167 us, total = 133.534 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:45:15,432 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 45128 total (25 active) Queueing time: mean = 15.716 us, max = 1.217 ms, min = 1.567 us, total = 709.230 ms Execution time: mean = 4.756 s, total = 214626.434 s Event stats: event_loop_lag_probe - 36708 total (0 active), Execution time: mean = 25.571 us, total = 938.677 ms, Queueing time: mean = 9.255 us, max = 1.017 ms, min = 1.567 us, total = 339.750 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2269 total (0 active), Execution time: mean = 36.983 us, total = 83.915 ms, Queueing time: mean = 84.367 us, max = 1.217 ms, min = 4.084 us, total = 191.429 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2269 total (24 active), Execution time: mean = 94.590 s, total = 214624.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1900 total (0 active), Execution time: mean = 102.684 us, total = 195.099 ms, Queueing time: mean = 91.542 us, max = 260.819 us, min = 11.233 us, total = 173.930 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1900 total (0 active), Execution time: mean = 555.138 us, total = 1.055 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 31 total (1 active), Execution time: mean = 441.088 us, total = 13.674 ms, Queueing time: mean = 68.681 us, max = 114.456 us, min = 33.142 us, total = 2.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:45:15,432 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 36754 total (0 active) Queueing time: mean = 9.748 us, max = 2.505 ms, min = 563.000 ns, total = 358.288 ms Execution time: mean = 25.676 us, total = 943.694 ms Event stats: event_loop_lag_probe - 36708 total (0 active), Execution time: mean = 25.608 us, total = 940.009 ms, Queueing time: mean = 9.691 us, max = 2.505 ms, min = 1.873 us, total = 355.732 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:46:15,432 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1843 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1912 - cumulative published bytes: 1521599 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:46:15,433 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 111047 total (4 active) Queueing time: mean = 58.118 us, max = 225.583 ms, min = -0.023 s, total = 6.454 s Execution time: mean = 226.612 us, total = 25.165 s Event stats: event_loop_lag_probe - 36947 total (0 active), Execution time: mean = 25.521 us, total = 942.939 ms, Queueing time: mean = 9.816 us, max = 6.895 ms, min = 1.268 us, total = 362.657 ms GcsInMemoryStore.Put - 14821 total (0 active), Execution time: mean = 20.673 us, total = 306.392 ms, Queueing time: mean = 149.987 us, max = 225.278 ms, min = 1.495 us, total = 2.223 s RayletLoadPulled - 9238 total (1 active), Execution time: mean = 244.818 us, total = 2.262 s, Queueing time: mean = 74.247 us, max = 3.337 ms, min = 10.621 us, total = 685.892 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9236 total (0 active), Execution time: mean = 75.141 us, total = 694.004 ms, Queueing time: mean = 99.845 us, max = 15.090 ms, min = 11.188 us, total = 922.169 ms NodeManagerService.grpc_client.GetResourceLoad - 9236 total (0 active), Execution time: mean = 1.378 ms, total = 12.729 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3701 total (0 active), Execution time: mean = 495.178 us, total = 1.833 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3701 total (0 active), Execution time: mean = 39.571 us, total = 146.452 ms, Queueing time: mean = 91.072 us, max = 601.063 us, min = 9.911 us, total = 337.056 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3694 total (0 active), Execution time: mean = 35.080 us, total = 129.585 ms, Queueing time: mean = 105.295 us, max = 1.496 ms, min = 13.520 us, total = 388.959 ms NodeInfoGcsService.grpc_server.CheckAlive - 3694 total (0 active), Execution time: mean = 515.354 us, total = 1.904 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3080 total (1 active), Execution time: mean = 10.079 us, total = 31.045 ms, Queueing time: mean = 74.416 us, max = 1.648 ms, min = 7.259 us, total = 229.201 ms HealthCheck - 3077 total (0 active), Execution time: mean = 7.643 us, total = 23.518 ms, Queueing time: mean = 91.218 us, max = 1.139 ms, min = 11.335 us, total = 280.677 ms GcsInMemoryStore.Get - 1903 total (0 active), Execution time: mean = 22.498 us, total = 42.815 ms, Queueing time: mean = 12.442 us, max = 137.899 us, min = 2.227 us, total = 23.677 ms InternalKVGcsService.grpc_server.InternalKVGet - 1902 total (0 active), Execution time: mean = 565.911 us, total = 1.076 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1902 total (0 active), Execution time: mean = 54.841 us, total = 104.307 ms, Queueing time: mean = 129.017 us, max = 22.982 ms, min = 9.184 us, total = 245.390 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1843 total (0 active), Execution time: mean = 616.215 us, total = 1.136 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1843 total (0 active), Execution time: mean = 90.713 us, total = 167.184 ms, Queueing time: mean = 104.944 us, max = 273.843 us, min = 15.511 us, total = 193.413 ms GCSServer.deadline_timer.debug_state_dump - 924 total (1 active), Execution time: mean = 1.365 ms, total = 1.261 s, Queueing time: mean = 43.155 us, max = 214.701 us, min = -0.023 s, total = 39.875 ms GCSServer.deadline_timer.debug_state_event_stats_print - 154 total (1 active, 1 running), Execution time: mean = 1.528 ms, total = 235.274 ms, Queueing time: mean = 59.459 us, max = 238.837 us, min = 13.943 us, total = 9.157 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:46:15,433 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 426709 total (1 active) Queueing time: mean = 40.916 us, max = 58.960 ms, min = 1.891 us, total = 17.459 s Execution time: mean = 218.135 us, total = 93.080 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 193956 total (0 active), Execution time: mean = 451.778 us, total = 87.625 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 193956 total (0 active), Execution time: mean = 23.165 us, total = 4.493 s, Queueing time: mean = 87.623 us, max = 58.960 ms, min = 2.005 us, total = 16.995 s event_loop_lag_probe - 36947 total (0 active), Execution time: mean = 25.565 us, total = 944.559 ms, Queueing time: mean = 8.924 us, max = 395.058 us, min = 1.891 us, total = 329.715 ms GcsTaskManager.GcJobSummary - 1849 total (1 active), Execution time: mean = 9.493 us, total = 17.553 ms, Queueing time: mean = 72.629 us, max = 3.642 ms, min = 12.167 us, total = 134.290 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:46:15,434 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 45416 total (25 active) Queueing time: mean = 15.709 us, max = 1.217 ms, min = 1.567 us, total = 713.431 ms Execution time: mean = 4.727 s, total = 214686.623 s Event stats: event_loop_lag_probe - 36948 total (0 active), Execution time: mean = 25.566 us, total = 944.616 ms, Queueing time: mean = 9.255 us, max = 1.017 ms, min = 1.567 us, total = 341.936 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2281 total (0 active), Execution time: mean = 36.981 us, total = 84.354 ms, Queueing time: mean = 84.345 us, max = 1.217 ms, min = 4.084 us, total = 192.391 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2281 total (24 active), Execution time: mean = 94.118 s, total = 214684.287 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1912 total (0 active), Execution time: mean = 102.637 us, total = 196.242 ms, Queueing time: mean = 91.519 us, max = 260.819 us, min = 11.233 us, total = 174.983 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1912 total (0 active), Execution time: mean = 554.841 us, total = 1.061 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 31 total (1 active), Execution time: mean = 441.088 us, total = 13.674 ms, Queueing time: mean = 68.681 us, max = 114.456 us, min = 33.142 us, total = 2.129 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:46:15,434 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 36994 total (0 active) Queueing time: mean = 9.746 us, max = 2.505 ms, min = 563.000 ns, total = 360.553 ms Execution time: mean = 25.671 us, total = 949.663 ms Event stats: event_loop_lag_probe - 36948 total (0 active), Execution time: mean = 25.603 us, total = 945.979 ms, Queueing time: mean = 9.689 us, max = 2.505 ms, min = 1.873 us, total = 357.996 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:47:15,434 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1855 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1924 - cumulative published bytes: 1531427 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:47:15,435 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 111765 total (4 active) Queueing time: mean = 58.007 us, max = 225.583 ms, min = -0.023 s, total = 6.483 s Execution time: mean = 226.559 us, total = 25.321 s Event stats: event_loop_lag_probe - 37186 total (0 active), Execution time: mean = 25.518 us, total = 948.919 ms, Queueing time: mean = 9.813 us, max = 6.895 ms, min = 1.268 us, total = 364.912 ms GcsInMemoryStore.Put - 14917 total (0 active), Execution time: mean = 20.574 us, total = 306.904 ms, Queueing time: mean = 149.282 us, max = 225.278 ms, min = 1.495 us, total = 2.227 s RayletLoadPulled - 9298 total (1 active), Execution time: mean = 244.851 us, total = 2.277 s, Queueing time: mean = 74.254 us, max = 3.337 ms, min = 10.621 us, total = 690.418 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9296 total (0 active), Execution time: mean = 75.113 us, total = 698.248 ms, Queueing time: mean = 100.237 us, max = 15.090 ms, min = 11.188 us, total = 931.800 ms NodeManagerService.grpc_client.GetResourceLoad - 9296 total (0 active), Execution time: mean = 1.379 ms, total = 12.815 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3725 total (0 active), Execution time: mean = 494.480 us, total = 1.842 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3725 total (0 active), Execution time: mean = 39.513 us, total = 147.188 ms, Queueing time: mean = 90.920 us, max = 601.063 us, min = 9.911 us, total = 338.679 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3718 total (0 active), Execution time: mean = 35.060 us, total = 130.352 ms, Queueing time: mean = 105.148 us, max = 1.496 ms, min = 13.520 us, total = 390.941 ms NodeInfoGcsService.grpc_server.CheckAlive - 3718 total (0 active), Execution time: mean = 515.019 us, total = 1.915 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3100 total (1 active), Execution time: mean = 10.072 us, total = 31.223 ms, Queueing time: mean = 74.467 us, max = 1.648 ms, min = 7.259 us, total = 230.848 ms HealthCheck - 3097 total (0 active), Execution time: mean = 7.640 us, total = 23.662 ms, Queueing time: mean = 91.054 us, max = 1.139 ms, min = 11.335 us, total = 281.994 ms GcsInMemoryStore.Get - 1915 total (0 active), Execution time: mean = 22.500 us, total = 43.087 ms, Queueing time: mean = 12.431 us, max = 137.899 us, min = 2.227 us, total = 23.806 ms InternalKVGcsService.grpc_server.InternalKVGet - 1914 total (0 active), Execution time: mean = 564.851 us, total = 1.081 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1914 total (0 active), Execution time: mean = 54.803 us, total = 104.892 ms, Queueing time: mean = 128.668 us, max = 22.982 ms, min = 9.184 us, total = 246.271 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1855 total (0 active), Execution time: mean = 615.462 us, total = 1.142 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1855 total (0 active), Execution time: mean = 90.715 us, total = 168.276 ms, Queueing time: mean = 104.781 us, max = 273.843 us, min = 15.511 us, total = 194.369 ms GCSServer.deadline_timer.debug_state_dump - 930 total (1 active), Execution time: mean = 1.366 ms, total = 1.270 s, Queueing time: mean = 43.303 us, max = 214.701 us, min = -0.023 s, total = 40.272 ms GCSServer.deadline_timer.debug_state_event_stats_print - 155 total (1 active, 1 running), Execution time: mean = 1.529 ms, total = 237.050 ms, Queueing time: mean = 59.570 us, max = 238.837 us, min = 13.943 us, total = 9.233 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:47:15,435 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 429481 total (1 active) Queueing time: mean = 40.867 us, max = 58.960 ms, min = 1.891 us, total = 17.552 s Execution time: mean = 217.998 us, total = 93.626 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 195216 total (0 active), Execution time: mean = 451.488 us, total = 88.138 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 195216 total (0 active), Execution time: mean = 23.154 us, total = 4.520 s, Queueing time: mean = 87.516 us, max = 58.960 ms, min = 2.005 us, total = 17.085 s event_loop_lag_probe - 37187 total (0 active), Execution time: mean = 25.563 us, total = 950.604 ms, Queueing time: mean = 8.924 us, max = 395.058 us, min = 1.891 us, total = 331.860 ms GcsTaskManager.GcJobSummary - 1861 total (1 active), Execution time: mean = 9.497 us, total = 17.674 ms, Queueing time: mean = 72.626 us, max = 3.642 ms, min = 12.167 us, total = 135.157 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:47:15,435 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 45715 total (25 active) Queueing time: mean = 15.712 us, max = 1.217 ms, min = 1.567 us, total = 718.259 ms Execution time: mean = 4.763 s, total = 217746.701 s Event stats: event_loop_lag_probe - 37188 total (0 active), Execution time: mean = 25.574 us, total = 951.057 ms, Queueing time: mean = 9.257 us, max = 1.017 ms, min = 1.567 us, total = 344.263 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2298 total (0 active), Execution time: mean = 36.957 us, total = 84.926 ms, Queueing time: mean = 84.361 us, max = 1.217 ms, min = 4.084 us, total = 193.861 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2298 total (24 active), Execution time: mean = 94.754 s, total = 217744.350 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1924 total (0 active), Execution time: mean = 102.727 us, total = 197.647 ms, Queueing time: mean = 91.413 us, max = 260.819 us, min = 11.233 us, total = 175.878 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1924 total (0 active), Execution time: mean = 554.680 us, total = 1.067 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 32 total (1 active), Execution time: mean = 431.095 us, total = 13.795 ms, Queueing time: mean = 70.819 us, max = 137.109 us, min = 33.142 us, total = 2.266 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:47:15,435 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 37234 total (0 active) Queueing time: mean = 9.748 us, max = 2.505 ms, min = 563.000 ns, total = 362.940 ms Execution time: mean = 25.676 us, total = 956.011 ms Event stats: event_loop_lag_probe - 37188 total (0 active), Execution time: mean = 25.608 us, total = 952.327 ms, Queueing time: mean = 9.691 us, max = 2.505 ms, min = 1.873 us, total = 360.384 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us [2025-01-21 07:48:15,436 I 16549 16549] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 25 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1867 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 536 - current buffered bytes: 0 RAY_LOG_CHANNEL - cumulative published messages: 1936 - cumulative published bytes: 1541255 - current buffered bytes: 819 GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 257 - current buffered bytes: 257 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 53 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 15 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 14 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 07:48:15,437 I 16549 16549] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 112484 total (4 active) Queueing time: mean = 57.902 us, max = 225.583 ms, min = -0.023 s, total = 6.513 s Execution time: mean = 226.677 us, total = 25.498 s Event stats: event_loop_lag_probe - 37426 total (0 active), Execution time: mean = 25.512 us, total = 954.824 ms, Queueing time: mean = 9.811 us, max = 6.895 ms, min = 1.268 us, total = 367.176 ms GcsInMemoryStore.Put - 15013 total (0 active), Execution time: mean = 20.482 us, total = 307.502 ms, Queueing time: mean = 148.610 us, max = 225.278 ms, min = 1.495 us, total = 2.231 s RayletLoadPulled - 9358 total (1 active), Execution time: mean = 244.939 us, total = 2.292 s, Queueing time: mean = 74.288 us, max = 3.337 ms, min = 10.621 us, total = 695.185 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 9356 total (0 active), Execution time: mean = 75.117 us, total = 702.792 ms, Queueing time: mean = 100.298 us, max = 15.090 ms, min = 11.188 us, total = 938.389 ms NodeManagerService.grpc_client.GetResourceLoad - 9356 total (0 active), Execution time: mean = 1.380 ms, total = 12.909 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut - 3749 total (0 active), Execution time: mean = 494.854 us, total = 1.855 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3749 total (0 active), Execution time: mean = 39.541 us, total = 148.239 ms, Queueing time: mean = 90.951 us, max = 601.063 us, min = 9.911 us, total = 340.975 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3742 total (0 active), Execution time: mean = 35.085 us, total = 131.288 ms, Queueing time: mean = 105.177 us, max = 1.496 ms, min = 13.520 us, total = 393.571 ms NodeInfoGcsService.grpc_server.CheckAlive - 3742 total (0 active), Execution time: mean = 515.313 us, total = 1.928 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 3120 total (1 active), Execution time: mean = 10.069 us, total = 31.417 ms, Queueing time: mean = 74.576 us, max = 1.648 ms, min = 7.259 us, total = 232.678 ms HealthCheck - 3117 total (0 active), Execution time: mean = 7.646 us, total = 23.833 ms, Queueing time: mean = 91.182 us, max = 1.139 ms, min = 11.335 us, total = 284.215 ms GcsInMemoryStore.Get - 1927 total (0 active), Execution time: mean = 22.531 us, total = 43.417 ms, Queueing time: mean = 12.437 us, max = 137.899 us, min = 2.227 us, total = 23.967 ms InternalKVGcsService.grpc_server.InternalKVGet - 1926 total (0 active), Execution time: mean = 564.926 us, total = 1.088 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1926 total (0 active), Execution time: mean = 54.817 us, total = 105.577 ms, Queueing time: mean = 128.449 us, max = 22.982 ms, min = 9.184 us, total = 247.394 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1867 total (0 active), Execution time: mean = 615.331 us, total = 1.149 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1867 total (0 active), Execution time: mean = 90.732 us, total = 169.396 ms, Queueing time: mean = 104.760 us, max = 273.843 us, min = 15.511 us, total = 195.586 ms GCSServer.deadline_timer.debug_state_dump - 936 total (1 active), Execution time: mean = 1.366 ms, total = 1.279 s, Queueing time: mean = 43.519 us, max = 214.701 us, min = -0.023 s, total = 40.734 ms GCSServer.deadline_timer.debug_state_event_stats_print - 156 total (1 active, 1 running), Execution time: mean = 1.530 ms, total = 238.653 ms, Queueing time: mean = 59.714 us, max = 238.837 us, min = 13.943 us, total = 9.315 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 25 total (0 active), Execution time: mean = 3.222 ms, total = 80.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 25 total (0 active), Execution time: mean = 976.134 us, total = 24.403 ms, Queueing time: mean = 1.887 ms, max = 22.652 ms, min = 13.842 us, total = 47.163 ms GcsResourceManager::Update - 22 total (0 active), Execution time: mean = 123.575 us, total = 2.719 ms, Queueing time: mean = 99.677 us, max = 175.233 us, min = 20.234 us, total = 2.193 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 488.802 us, total = 10.265 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 27.527 us, total = 578.066 us, Queueing time: mean = 119.221 us, max = 990.002 us, min = 13.763 us, total = 2.504 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.811 us, total = 52.865 us, Queueing time: mean = 33.528 us, max = 40.516 us, min = 10.559 us, total = 201.166 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 801.096 us, total = 4.005 ms, Queueing time: mean = 170.426 us, max = 455.845 us, min = 90.434 us, total = 852.129 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 2.499 ms, total = 12.495 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 138.179 us, total = 552.714 us, Queueing time: mean = 114.822 ms, max = 225.583 ms, min = 3.931 ms, total = 459.287 ms InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 669.950 us, total = 669.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 18.185 us, total = 18.185 us, Queueing time: mean = 727.000 ns, max = 727.000 ns, min = 727.000 ns, total = 727.000 ns JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 493.580 us, total = 493.580 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 16.907 us, total = 16.907 us, Queueing time: mean = 10.800 us, max = 10.800 us, min = 10.800 us, total = 10.800 us NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 176.392 us, total = 176.392 us, Queueing time: mean = 87.087 us, max = 87.087 us, min = 87.087 us, total = 87.087 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 476.372 us, total = 476.372 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.867 us, total = 35.867 us, Queueing time: mean = 42.367 us, max = 42.367 us, min = 42.367 us, total = 42.367 us GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 9.652 us, total = 9.652 us, Queueing time: mean = 7.678 us, max = 7.678 us, min = 7.678 us, total = 7.678 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 137.810 us, total = 137.810 us, Queueing time: mean = 45.765 us, max = 45.765 us, min = 45.765 us, total = 45.765 us InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 77.259 us, total = 77.259 us, Queueing time: mean = 90.912 us, max = 90.912 us, min = 90.912 us, total = 90.912 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.304 us, total = 22.304 us, Queueing time: mean = 90.923 us, max = 90.923 us, min = 90.923 us, total = 90.923 us InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 621.211 us, total = 621.211 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 660.939 us, total = 660.939 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 483.254 us, total = 483.254 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.277 us, total = 22.277 us, Queueing time: mean = 96.823 us, max = 96.823 us, min = 96.823 us, total = 96.823 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.036 ms, total = 1.036 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 43.799 us, total = 43.799 us, Queueing time: mean = 92.967 us, max = 92.967 us, min = 92.967 us, total = 92.967 us [2025-01-21 07:48:15,437 I 16549 16549] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 432253 total (1 active) Queueing time: mean = 40.932 us, max = 58.960 ms, min = 1.891 us, total = 17.693 s Execution time: mean = 218.302 us, total = 94.362 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 196476 total (0 active), Execution time: mean = 452.123 us, total = 88.831 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 196476 total (0 active), Execution time: mean = 23.189 us, total = 4.556 s, Queueing time: mean = 87.658 us, max = 58.960 ms, min = 2.005 us, total = 17.223 s event_loop_lag_probe - 37427 total (0 active), Execution time: mean = 25.563 us, total = 956.762 ms, Queueing time: mean = 8.923 us, max = 395.058 us, min = 1.891 us, total = 333.967 ms GcsTaskManager.GcJobSummary - 1873 total (1 active), Execution time: mean = 9.500 us, total = 17.793 ms, Queueing time: mean = 72.747 us, max = 3.642 ms, min = 12.167 us, total = 136.256 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.305 us, total = 11.305 us, Queueing time: mean = 15.917 us, max = 15.917 us, min = 15.917 us, total = 15.917 us [2025-01-21 07:48:15,437 I 16549 16549] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 46003 total (25 active) Queueing time: mean = 15.721 us, max = 1.217 ms, min = 1.567 us, total = 723.204 ms Execution time: mean = 4.735 s, total = 217806.875 s Event stats: event_loop_lag_probe - 37428 total (0 active), Execution time: mean = 25.579 us, total = 957.382 ms, Queueing time: mean = 9.261 us, max = 1.017 ms, min = 1.567 us, total = 346.626 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2310 total (0 active), Execution time: mean = 37.077 us, total = 85.648 ms, Queueing time: mean = 84.407 us, max = 1.217 ms, min = 4.084 us, total = 194.980 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2310 total (24 active), Execution time: mean = 94.288 s, total = 217804.508 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1936 total (0 active), Execution time: mean = 102.740 us, total = 198.905 ms, Queueing time: mean = 91.602 us, max = 260.819 us, min = 11.233 us, total = 177.341 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1936 total (0 active), Execution time: mean = 555.366 us, total = 1.075 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 32 total (1 active), Execution time: mean = 431.095 us, total = 13.795 ms, Queueing time: mean = 70.819 us, max = 137.109 us, min = 33.142 us, total = 2.266 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 45.227 us, total = 1.131 ms, Queueing time: mean = 79.114 us, max = 136.001 us, min = 12.468 us, total = 1.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 1.400 ms, total = 35.006 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 10.862 us, total = 10.862 us, Queueing time: mean = 13.354 us, max = 13.354 us, min = 13.354 us, total = 13.354 us [2025-01-21 07:48:15,437 I 16549 16549] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 37474 total (0 active) Queueing time: mean = 9.751 us, max = 2.505 ms, min = 563.000 ns, total = 365.392 ms Execution time: mean = 25.681 us, total = 962.370 ms Event stats: event_loop_lag_probe - 37428 total (0 active), Execution time: mean = 25.614 us, total = 958.686 ms, Queueing time: mean = 9.694 us, max = 2.505 ms, min = 1.873 us, total = 362.836 ms - 22 total (0 active), Execution time: mean = 134.808 us, total = 2.966 ms, Queueing time: mean = 103.329 us, max = 205.168 us, min = 15.988 us, total = 2.273 ms RaySyncer.BroadcastMessage - 22 total (0 active), Execution time: mean = 32.608 us, total = 717.386 us, Queueing time: mean = 955.364 ns, max = 1.380 us, min = 563.000 ns, total = 21.018 us RaySyncerRegister - 2 total (0 active), Execution time: mean = 550.000 ns, total = 1.100 us, Queueing time: mean = 130.940 us, max = 131.791 us, min = 130.088 us, total = 261.879 us