[2025-01-20 23:01:53,505 I 13457 13457] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed [2025-01-20 23:01:53,506 I 13457 13457] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. [2025-01-20 23:01:53,511 I 13457 13457] (gcs_server) event.cc:493: Ray Event initialized for GCS [2025-01-20 23:01:53,511 I 13457 13457] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE [2025-01-20 23:01:53,511 I 13457 13457] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR [2025-01-20 23:01:53,511 I 13457 13457] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB [2025-01-20 23:01:53,511 I 13457 13457] (gcs_server) event.cc:324: Set ray event level to warning [2025-01-20 23:01:53,515 I 13457 13457] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:42: Loading job table data. [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:54: Loading node table data. [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:80: Loading actor table data. [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 [2025-01-20 23:01:53,519 I 13457 13457] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 7cd267f3e9ae7d263565d743c4adec3db9f4c00e2d94b48fe89053a6 [2025-01-20 23:01:53,520 I 13457 13457] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 [2025-01-20 23:01:53,523 I 13457 13457] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 63200. [2025-01-20 23:01:53,803 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 0 - DrainNode request count: 0 - GetAllNodeInfo request count: 0 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 0 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 0 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 0 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 0 -Total num of driver tasks: 0 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:01:53,803 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 25 total (5 active) Queueing time: mean = 100.863 ms, max = 279.303 ms, min = 3.444 us, total = 2.522 s Execution time: mean = 11.345 ms, total = 283.630 ms Event stats: GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 31.038 ms, total = 279.341 ms, Queueing time: mean = 216.447 ms, max = 278.545 ms, min = 3.444 us, total = 1.948 s GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 13.410 us, total = 67.052 us, Queueing time: mean = 88.742 us, max = 101.411 us, min = 78.379 us, total = 443.708 us PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 4.005 us, total = 16.020 us, Queueing time: mean = 139.607 ms, max = 279.303 ms, min = 279.126 ms, total = 558.429 ms event_loop_lag_probe - 2 total (0 active), Execution time: mean = 12.931 us, total = 25.863 us, Queueing time: mean = 6.764 ms, max = 13.097 ms, min = 432.010 us, total = 13.529 ms GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 22.123 us, total = 22.123 us, Queueing time: mean = 5.167 us, max = 5.167 us, min = 5.167 us, total = 5.167 us NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 4.159 ms, total = 4.159 ms, Queueing time: mean = 1.136 ms, max = 1.136 ms, min = 1.136 ms, total = 1.136 ms [2025-01-20 23:01:53,803 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 5 total (1 active) Queueing time: mean = 150.817 us, max = 618.612 us, min = 9.205 us, total = 754.083 us Execution time: mean = 156.166 us, total = 780.829 us Event stats: event_loop_lag_probe - 3 total (0 active), Execution time: mean = 257.398 us, total = 772.193 us, Queueing time: mean = 218.031 us, max = 618.612 us, min = 9.205 us, total = 654.094 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:01:53,803 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5 total (1 active) Queueing time: mean = 87.933 us, max = 291.078 us, min = 11.625 us, total = 439.664 us Execution time: mean = 55.071 us, total = 275.357 us Event stats: event_loop_lag_probe - 3 total (0 active), Execution time: mean = 85.614 us, total = 256.841 us, Queueing time: mean = 110.518 us, max = 291.078 us, min = 11.625 us, total = 331.555 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:01:53,803 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5 total (0 active) Queueing time: mean = 343.594 us, max = 1.423 ms, min = 9.242 us, total = 1.718 ms Execution time: mean = 50.663 us, total = 253.316 us Event stats: event_loop_lag_probe - 3 total (0 active), Execution time: mean = 83.699 us, total = 251.096 us, Queueing time: mean = 488.948 us, max = 1.423 ms, min = 9.242 us, total = 1.467 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:01:55,921 I 13457 13457] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=94df2f5134fc51771c1dcb24cd489e24913981486355e9608be28de3 [2025-01-20 23:01:55,921 I 13457 13457] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=94df2f5134fc51771c1dcb24cd489e24913981486355e9608be28de3 [2025-01-20 23:01:55,921 I 13457 13457] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 94df2f5134fc51771c1dcb24cd489e24913981486355e9608be28de3 registered, will try to reschedule all the infeasible placement groups. [2025-01-20 23:01:55,928 I 13457 13512] (gcs_server) ray_syncer.cc:377: Get connection node_id=94df2f5134fc51771c1dcb24cd489e24913981486355e9608be28de3 [2025-01-20 23:01:56,926 I 13457 13457] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 8700 [2025-01-20 23:01:56,926 I 13457 13457] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 8700 [2025-01-20 23:02:03,525 W 13457 13459] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:64875: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster. [2025-01-20 23:02:53,804 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 12 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 22 - cumulative published bytes: 13652 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 39 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:02:53,805 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 1151 total (4 active) Queueing time: mean = 2.243 ms, max = 279.303 ms, min = 807.000 ns, total = 2.582 s Execution time: mean = 492.738 us, total = 567.142 ms Event stats: event_loop_lag_probe - 242 total (0 active), Execution time: mean = 26.594 us, total = 6.436 ms, Queueing time: mean = 66.113 us, max = 13.097 ms, min = 3.649 us, total = 15.999 ms GcsInMemoryStore.Put - 142 total (0 active), Execution time: mean = 1.981 ms, total = 281.231 ms, Queueing time: mean = 13.755 ms, max = 278.545 ms, min = 2.310 us, total = 1.953 s GcsInMemoryStore.Get - 81 total (0 active), Execution time: mean = 26.381 us, total = 2.137 ms, Queueing time: mean = 16.196 us, max = 64.163 us, min = 4.194 us, total = 1.312 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 80 total (0 active), Execution time: mean = 44.031 us, total = 3.522 ms, Queueing time: mean = 117.583 us, max = 269.544 us, min = 13.883 us, total = 9.407 ms InternalKVGcsService.grpc_server.InternalKVGet - 80 total (0 active), Execution time: mean = 644.231 us, total = 51.538 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s RayletLoadPulled - 60 total (1 active), Execution time: mean = 238.954 us, total = 14.337 ms, Queueing time: mean = 71.718 us, max = 149.398 us, min = 20.372 us, total = 4.303 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 75.802 us, total = 4.321 ms, Queueing time: mean = 111.204 us, max = 269.272 us, min = 21.635 us, total = 6.339 ms NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.519 ms, total = 86.578 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 38 total (0 active), Execution time: mean = 45.092 us, total = 1.713 ms, Queueing time: mean = 107.282 us, max = 193.151 us, min = 15.247 us, total = 4.077 ms InternalKVGcsService.grpc_server.InternalKVPut - 38 total (0 active), Execution time: mean = 608.461 us, total = 23.122 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 33.342 us, total = 800.216 us, Queueing time: mean = 117.144 us, max = 162.838 us, min = 40.598 us, total = 2.811 ms NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 607.695 us, total = 14.585 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 21 total (1 active), Execution time: mean = 9.294 us, total = 195.177 us, Queueing time: mean = 78.656 us, max = 152.814 us, min = 29.516 us, total = 1.652 ms HealthCheck - 18 total (0 active), Execution time: mean = 9.168 us, total = 165.026 us, Queueing time: mean = 101.840 us, max = 172.403 us, min = 34.906 us, total = 1.833 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 677.097 us, total = 8.125 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 93.427 us, total = 1.121 ms, Queueing time: mean = 111.138 us, max = 156.025 us, min = 24.786 us, total = 1.334 ms GcsResourceManager::Update - 11 total (0 active), Execution time: mean = 113.320 us, total = 1.247 ms, Queueing time: mean = 139.552 us, max = 191.304 us, min = 21.005 us, total = 1.535 ms GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.200 ms, total = 7.199 ms, Queueing time: mean = 97.430 us, max = 315.299 us, min = 49.417 us, total = 584.583 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:02:53,805 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 2667 total (1 active) Queueing time: mean = 53.242 us, max = 618.612 us, min = 4.294 us, total = 141.996 ms Execution time: mean = 267.203 us, total = 712.631 ms Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 1205 total (0 active), Execution time: mean = 555.099 us, total = 668.894 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1205 total (0 active), Execution time: mean = 30.032 us, total = 36.188 ms, Queueing time: mean = 114.371 us, max = 242.366 us, min = 7.319 us, total = 137.817 ms event_loop_lag_probe - 243 total (0 active), Execution time: mean = 30.372 us, total = 7.380 ms, Queueing time: mean = 12.340 us, max = 618.612 us, min = 4.294 us, total = 2.999 ms GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 12.335 us, total = 160.358 us, Queueing time: mean = 83.158 us, max = 110.869 us, min = 64.059 us, total = 1.081 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:02:53,805 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 431 total (25 active) Queueing time: mean = 35.040 us, max = 333.684 us, min = 2.330 us, total = 15.102 ms Execution time: mean = 124.546 ms, total = 53.679 s Event stats: event_loop_lag_probe - 243 total (0 active), Execution time: mean = 30.264 us, total = 7.354 ms, Queueing time: mean = 11.668 us, max = 291.078 us, min = 2.330 us, total = 2.835 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 46 total (0 active), Execution time: mean = 34.339 us, total = 1.580 ms, Queueing time: mean = 136.141 us, max = 333.684 us, min = 14.442 us, total = 6.262 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 46 total (24 active), Execution time: mean = 1.166 s, total = 53.637 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 22 total (0 active), Execution time: mean = 113.947 us, total = 2.507 ms, Queueing time: mean = 117.594 us, max = 212.157 us, min = 31.977 us, total = 2.587 ms InternalPubSubGcsService.grpc_server.GcsPublish - 22 total (0 active), Execution time: mean = 652.665 us, total = 14.359 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:02:53,805 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 267 total (0 active) Queueing time: mean = 20.119 us, max = 1.423 ms, min = 568.000 ns, total = 5.372 ms Execution time: mean = 28.853 us, total = 7.704 ms Event stats: event_loop_lag_probe - 243 total (0 active), Execution time: mean = 25.963 us, total = 6.309 ms, Queueing time: mean = 15.001 us, max = 1.423 ms, min = 1.850 us, total = 3.645 ms RaySyncer.BroadcastMessage - 11 total (0 active), Execution time: mean = 27.532 us, total = 302.853 us, Queueing time: mean = 1.153 us, max = 1.836 us, min = 568.000 ns, total = 12.687 us - 11 total (0 active), Execution time: mean = 99.044 us, total = 1.089 ms, Queueing time: mean = 132.967 us, max = 163.613 us, min = 33.336 us, total = 1.463 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:03:53,806 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 24 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 35 - cumulative published bytes: 23501 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:03:53,806 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 1871 total (4 active) Queueing time: mean = 1.397 ms, max = 279.303 ms, min = 807.000 ns, total = 2.614 s Execution time: mean = 396.785 us, total = 742.384 ms Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.862 us, total = 12.465 ms, Queueing time: mean = 38.130 us, max = 13.097 ms, min = 3.649 us, total = 18.379 ms GcsInMemoryStore.Put - 238 total (0 active), Execution time: mean = 1.184 ms, total = 281.819 ms, Queueing time: mean = 8.225 ms, max = 278.545 ms, min = 2.310 us, total = 1.958 s RayletLoadPulled - 120 total (1 active), Execution time: mean = 242.917 us, total = 29.150 ms, Queueing time: mean = 76.367 us, max = 149.398 us, min = 20.372 us, total = 9.164 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 117 total (0 active), Execution time: mean = 76.703 us, total = 8.974 ms, Queueing time: mean = 109.763 us, max = 269.272 us, min = 21.635 us, total = 12.842 ms NodeManagerService.grpc_client.GetResourceLoad - 117 total (0 active), Execution time: mean = 1.513 ms, total = 176.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 93 total (0 active), Execution time: mean = 25.746 us, total = 2.394 ms, Queueing time: mean = 15.889 us, max = 64.163 us, min = 4.194 us, total = 1.478 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 92 total (0 active), Execution time: mean = 45.330 us, total = 4.170 ms, Queueing time: mean = 117.208 us, max = 269.544 us, min = 13.883 us, total = 10.783 ms InternalKVGcsService.grpc_server.InternalKVGet - 92 total (0 active), Execution time: mean = 635.314 us, total = 58.449 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 62 total (0 active), Execution time: mean = 43.496 us, total = 2.697 ms, Queueing time: mean = 109.203 us, max = 193.151 us, min = 15.247 us, total = 6.771 ms InternalKVGcsService.grpc_server.InternalKVPut - 62 total (0 active), Execution time: mean = 599.822 us, total = 37.189 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 32.149 us, total = 1.543 ms, Queueing time: mean = 123.185 us, max = 202.100 us, min = 40.598 us, total = 5.913 ms NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 605.780 us, total = 29.077 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 41 total (1 active), Execution time: mean = 9.431 us, total = 386.667 us, Queueing time: mean = 79.796 us, max = 152.814 us, min = 29.516 us, total = 3.272 ms HealthCheck - 38 total (0 active), Execution time: mean = 8.757 us, total = 332.769 us, Queueing time: mean = 112.140 us, max = 219.229 us, min = 28.329 us, total = 4.261 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 716.473 us, total = 17.195 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 92.186 us, total = 2.212 ms, Queueing time: mean = 125.970 us, max = 190.124 us, min = 24.786 us, total = 3.023 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.292 ms, total = 15.509 ms, Queueing time: mean = 79.960 us, max = 315.299 us, min = 48.782 us, total = 959.516 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 871.456 us, total = 1.743 ms, Queueing time: mean = 36.901 us, max = 73.803 us, min = 73.803 us, total = 73.803 us GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:03:53,807 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 5428 total (1 active) Queueing time: mean = 51.864 us, max = 618.612 us, min = 3.712 us, total = 281.518 ms Execution time: mean = 265.117 us, total = 1.439 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 2460 total (0 active), Execution time: mean = 549.665 us, total = 1.352 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2460 total (0 active), Execution time: mean = 29.477 us, total = 72.514 ms, Queueing time: mean = 111.385 us, max = 242.366 us, min = 7.009 us, total = 274.008 ms event_loop_lag_probe - 482 total (0 active), Execution time: mean = 29.118 us, total = 14.035 ms, Queueing time: mean = 10.953 us, max = 618.612 us, min = 3.712 us, total = 5.279 ms GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 12.935 us, total = 323.368 us, Queueing time: mean = 85.258 us, max = 111.670 us, min = 59.022 us, total = 2.131 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:03:53,807 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 722 total (25 active) Queueing time: mean = 29.163 us, max = 333.684 us, min = 2.330 us, total = 21.056 ms Execution time: mean = 157.780 ms, total = 113.917 s Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 31.131 us, total = 15.005 ms, Queueing time: mean = 11.887 us, max = 291.078 us, min = 2.330 us, total = 5.729 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 59 total (0 active), Execution time: mean = 39.156 us, total = 2.310 ms, Queueing time: mean = 131.589 us, max = 333.684 us, min = 14.442 us, total = 7.764 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 59 total (24 active), Execution time: mean = 1.930 s, total = 113.856 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 35 total (0 active), Execution time: mean = 114.440 us, total = 4.005 ms, Queueing time: mean = 118.440 us, max = 212.157 us, min = 31.977 us, total = 4.145 ms InternalPubSubGcsService.grpc_server.GcsPublish - 35 total (0 active), Execution time: mean = 652.767 us, total = 22.847 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:03:53,807 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 508 total (0 active) Queueing time: mean = 15.559 us, max = 1.423 ms, min = 568.000 ns, total = 7.904 ms Execution time: mean = 27.847 us, total = 14.146 ms Event stats: event_loop_lag_probe - 482 total (0 active), Execution time: mean = 26.136 us, total = 12.597 ms, Queueing time: mean = 12.500 us, max = 1.423 ms, min = 1.850 us, total = 6.025 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:04:53,807 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 36 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 48 - cumulative published bytes: 33350 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:04:53,808 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 2590 total (4 active) Queueing time: mean = 1.021 ms, max = 279.303 ms, min = 807.000 ns, total = 2.645 s Execution time: mean = 353.574 us, total = 915.757 ms Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.499 us, total = 18.411 ms, Queueing time: mean = 28.661 us, max = 13.097 ms, min = 3.649 us, total = 20.693 ms GcsInMemoryStore.Put - 334 total (0 active), Execution time: mean = 845.440 us, total = 282.377 ms, Queueing time: mean = 5.878 ms, max = 278.545 ms, min = 2.310 us, total = 1.963 s RayletLoadPulled - 180 total (1 active), Execution time: mean = 237.085 us, total = 42.675 ms, Queueing time: mean = 74.689 us, max = 149.398 us, min = 20.372 us, total = 13.444 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 177 total (0 active), Execution time: mean = 75.494 us, total = 13.363 ms, Queueing time: mean = 108.471 us, max = 269.272 us, min = 21.635 us, total = 19.199 ms NodeManagerService.grpc_client.GetResourceLoad - 177 total (0 active), Execution time: mean = 1.495 ms, total = 264.652 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 105 total (0 active), Execution time: mean = 25.147 us, total = 2.640 ms, Queueing time: mean = 15.598 us, max = 64.163 us, min = 4.194 us, total = 1.638 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 104 total (0 active), Execution time: mean = 45.973 us, total = 4.781 ms, Queueing time: mean = 116.298 us, max = 269.544 us, min = 13.883 us, total = 12.095 ms InternalKVGcsService.grpc_server.InternalKVGet - 104 total (0 active), Execution time: mean = 631.648 us, total = 65.691 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 86 total (0 active), Execution time: mean = 41.856 us, total = 3.600 ms, Queueing time: mean = 108.669 us, max = 193.151 us, min = 15.247 us, total = 9.346 ms InternalKVGcsService.grpc_server.InternalKVPut - 86 total (0 active), Execution time: mean = 597.419 us, total = 51.378 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 31.339 us, total = 2.256 ms, Queueing time: mean = 121.971 us, max = 203.410 us, min = 26.210 us, total = 8.782 ms NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 602.589 us, total = 43.386 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 61 total (1 active), Execution time: mean = 9.735 us, total = 593.852 us, Queueing time: mean = 80.005 us, max = 152.814 us, min = 29.516 us, total = 4.880 ms HealthCheck - 58 total (0 active), Execution time: mean = 8.426 us, total = 488.716 us, Queueing time: mean = 112.653 us, max = 219.229 us, min = 25.661 us, total = 6.534 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 90.592 us, total = 3.261 ms, Queueing time: mean = 121.290 us, max = 190.124 us, min = 24.786 us, total = 4.366 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 818.182 us, total = 29.455 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.297 ms, total = 23.350 ms, Queueing time: mean = 74.600 us, max = 315.299 us, min = 30.033 us, total = 1.343 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 1.097 ms, total = 3.290 ms, Queueing time: mean = 45.520 us, max = 73.803 us, min = 62.757 us, total = 136.560 us GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:04:53,808 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 8194 total (1 active) Queueing time: mean = 52.091 us, max = 618.612 us, min = 3.712 us, total = 426.836 ms Execution time: mean = 265.657 us, total = 2.177 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 3717 total (0 active), Execution time: mean = 551.374 us, total = 2.049 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3717 total (0 active), Execution time: mean = 28.688 us, total = 106.633 ms, Queueing time: mean = 111.975 us, max = 301.275 us, min = 7.009 us, total = 416.211 ms event_loop_lag_probe - 722 total (0 active), Execution time: mean = 28.020 us, total = 20.230 ms, Queueing time: mean = 10.352 us, max = 618.612 us, min = 3.712 us, total = 7.474 ms GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 12.534 us, total = 463.760 us, Queueing time: mean = 82.456 us, max = 114.655 us, min = 53.375 us, total = 3.051 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:04:53,808 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1014 total (25 active) Queueing time: mean = 26.519 us, max = 333.684 us, min = 2.330 us, total = 26.891 ms Execution time: mean = 171.694 ms, total = 174.098 s Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 30.426 us, total = 21.968 ms, Queueing time: mean = 11.482 us, max = 291.078 us, min = 2.330 us, total = 8.290 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 72 total (0 active), Execution time: mean = 40.010 us, total = 2.881 ms, Queueing time: mean = 129.131 us, max = 333.684 us, min = 14.442 us, total = 9.297 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 72 total (24 active), Execution time: mean = 2.417 s, total = 174.019 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 48 total (0 active), Execution time: mean = 112.926 us, total = 5.420 ms, Queueing time: mean = 122.626 us, max = 212.157 us, min = 31.977 us, total = 5.886 ms InternalPubSubGcsService.grpc_server.GcsPublish - 48 total (0 active), Execution time: mean = 660.306 us, total = 31.695 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:04:53,808 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 748 total (0 active) Queueing time: mean = 14.073 us, max = 1.423 ms, min = 568.000 ns, total = 10.526 ms Execution time: mean = 28.219 us, total = 21.108 ms Event stats: event_loop_lag_probe - 722 total (0 active), Execution time: mean = 27.089 us, total = 19.558 ms, Queueing time: mean = 11.977 us, max = 1.423 ms, min = 1.850 us, total = 8.647 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:05:53,808 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 48 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 60 - cumulative published bytes: 43166 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:05:53,809 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 3309 total (4 active) Queueing time: mean = 808.334 us, max = 279.303 ms, min = 807.000 ns, total = 2.675 s Execution time: mean = 327.704 us, total = 1.084 s Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 25.221 us, total = 24.263 ms, Queueing time: mean = 23.818 us, max = 13.097 ms, min = 3.649 us, total = 22.913 ms GcsInMemoryStore.Put - 430 total (0 active), Execution time: mean = 657.869 us, total = 282.884 ms, Queueing time: mean = 4.575 ms, max = 278.545 ms, min = 2.310 us, total = 1.967 s RayletLoadPulled - 240 total (1 active), Execution time: mean = 233.717 us, total = 56.092 ms, Queueing time: mean = 73.852 us, max = 149.398 us, min = 19.506 us, total = 17.724 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 237 total (0 active), Execution time: mean = 75.147 us, total = 17.810 ms, Queueing time: mean = 108.388 us, max = 269.272 us, min = 21.635 us, total = 25.688 ms NodeManagerService.grpc_client.GetResourceLoad - 237 total (0 active), Execution time: mean = 1.486 ms, total = 352.261 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 117 total (0 active), Execution time: mean = 24.761 us, total = 2.897 ms, Queueing time: mean = 15.385 us, max = 64.163 us, min = 4.194 us, total = 1.800 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 116 total (0 active), Execution time: mean = 46.860 us, total = 5.436 ms, Queueing time: mean = 116.054 us, max = 269.544 us, min = 13.883 us, total = 13.462 ms InternalKVGcsService.grpc_server.InternalKVGet - 116 total (0 active), Execution time: mean = 629.255 us, total = 72.994 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 110 total (0 active), Execution time: mean = 41.252 us, total = 4.538 ms, Queueing time: mean = 108.582 us, max = 193.151 us, min = 15.247 us, total = 11.944 ms InternalKVGcsService.grpc_server.InternalKVPut - 110 total (0 active), Execution time: mean = 592.285 us, total = 65.151 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 30.762 us, total = 2.953 ms, Queueing time: mean = 124.038 us, max = 212.853 us, min = 26.210 us, total = 11.908 ms NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 608.316 us, total = 58.398 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 81 total (1 active), Execution time: mean = 9.756 us, total = 790.219 us, Queueing time: mean = 80.856 us, max = 152.814 us, min = 29.516 us, total = 6.549 ms HealthCheck - 78 total (0 active), Execution time: mean = 8.200 us, total = 639.580 us, Queueing time: mean = 109.334 us, max = 219.229 us, min = 23.146 us, total = 8.528 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 86.891 us, total = 4.171 ms, Queueing time: mean = 117.927 us, max = 190.124 us, min = 24.786 us, total = 5.661 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 774.663 us, total = 37.184 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.314 ms, total = 31.535 ms, Queueing time: mean = 77.745 us, max = 315.299 us, min = 30.033 us, total = 1.866 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.067 ms, total = 4.267 ms, Queueing time: mean = 42.992 us, max = 73.803 us, min = 35.406 us, total = 171.966 us GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:05:53,809 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 10966 total (1 active) Queueing time: mean = 52.265 us, max = 618.612 us, min = 3.450 us, total = 573.141 ms Execution time: mean = 266.170 us, total = 2.919 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 4977 total (0 active), Execution time: mean = 552.776 us, total = 2.751 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4977 total (0 active), Execution time: mean = 28.177 us, total = 140.239 ms, Queueing time: mean = 112.376 us, max = 301.275 us, min = 7.009 us, total = 559.295 ms event_loop_lag_probe - 962 total (0 active), Execution time: mean = 27.870 us, total = 26.811 ms, Queueing time: mean = 10.173 us, max = 618.612 us, min = 3.450 us, total = 9.787 ms GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 12.086 us, total = 592.206 us, Queueing time: mean = 80.795 us, max = 153.496 us, min = 27.203 us, total = 3.959 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:05:53,810 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1302 total (25 active) Queueing time: mean = 24.839 us, max = 333.684 us, min = 2.330 us, total = 32.340 ms Execution time: mean = 179.948 ms, total = 234.292 s Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 29.829 us, total = 28.695 ms, Queueing time: mean = 11.194 us, max = 291.078 us, min = 2.330 us, total = 10.769 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 84 total (0 active), Execution time: mean = 42.629 us, total = 3.581 ms, Queueing time: mean = 127.709 us, max = 333.684 us, min = 14.442 us, total = 10.728 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 84 total (24 active), Execution time: mean = 2.788 s, total = 234.196 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 60 total (0 active), Execution time: mean = 117.001 us, total = 7.020 ms, Queueing time: mean = 123.777 us, max = 212.157 us, min = 31.977 us, total = 7.427 ms InternalPubSubGcsService.grpc_server.GcsPublish - 60 total (0 active), Execution time: mean = 677.623 us, total = 40.657 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s [2025-01-20 23:05:53,810 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 988 total (0 active) Queueing time: mean = 13.769 us, max = 1.423 ms, min = 568.000 ns, total = 13.603 ms Execution time: mean = 29.636 us, total = 29.280 ms Event stats: event_loop_lag_probe - 962 total (0 active), Execution time: mean = 28.827 us, total = 27.731 ms, Queueing time: mean = 12.187 us, max = 1.423 ms, min = 1.850 us, total = 11.724 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:06:53,810 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 60 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 72 - cumulative published bytes: 52982 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:06:53,811 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 4028 total (4 active) Queueing time: mean = 671.737 us, max = 279.303 ms, min = 807.000 ns, total = 2.706 s Execution time: mean = 311.391 us, total = 1.254 s Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 25.570 us, total = 30.735 ms, Queueing time: mean = 21.188 us, max = 13.097 ms, min = 3.649 us, total = 25.468 ms GcsInMemoryStore.Put - 526 total (0 active), Execution time: mean = 538.855 us, total = 283.438 ms, Queueing time: mean = 3.750 ms, max = 278.545 ms, min = 2.310 us, total = 1.973 s RayletLoadPulled - 300 total (1 active), Execution time: mean = 234.490 us, total = 70.347 ms, Queueing time: mean = 73.679 us, max = 161.851 us, min = 19.183 us, total = 22.104 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 297 total (0 active), Execution time: mean = 75.213 us, total = 22.338 ms, Queueing time: mean = 108.441 us, max = 269.272 us, min = 18.998 us, total = 32.207 ms NodeManagerService.grpc_client.GetResourceLoad - 297 total (0 active), Execution time: mean = 1.482 ms, total = 440.063 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 134 total (0 active), Execution time: mean = 40.523 us, total = 5.430 ms, Queueing time: mean = 106.872 us, max = 193.151 us, min = 15.181 us, total = 14.321 ms InternalKVGcsService.grpc_server.InternalKVPut - 134 total (0 active), Execution time: mean = 583.962 us, total = 78.251 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 129 total (0 active), Execution time: mean = 24.602 us, total = 3.174 ms, Queueing time: mean = 15.174 us, max = 64.163 us, min = 4.194 us, total = 1.957 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 128 total (0 active), Execution time: mean = 47.596 us, total = 6.092 ms, Queueing time: mean = 115.069 us, max = 269.544 us, min = 13.883 us, total = 14.729 ms InternalKVGcsService.grpc_server.InternalKVGet - 128 total (0 active), Execution time: mean = 624.562 us, total = 79.944 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 120 total (0 active), Execution time: mean = 30.876 us, total = 3.705 ms, Queueing time: mean = 120.597 us, max = 212.853 us, min = 26.210 us, total = 14.472 ms NodeInfoGcsService.grpc_server.CheckAlive - 120 total (0 active), Execution time: mean = 601.220 us, total = 72.146 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 101 total (1 active), Execution time: mean = 10.136 us, total = 1.024 ms, Queueing time: mean = 82.361 us, max = 154.987 us, min = 29.516 us, total = 8.318 ms HealthCheck - 98 total (0 active), Execution time: mean = 8.213 us, total = 804.861 us, Queueing time: mean = 107.161 us, max = 219.229 us, min = 20.526 us, total = 10.502 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 60 total (0 active), Execution time: mean = 87.297 us, total = 5.238 ms, Queueing time: mean = 116.617 us, max = 190.124 us, min = 24.786 us, total = 6.997 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 60 total (0 active), Execution time: mean = 751.597 us, total = 45.096 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_dump - 30 total (1 active), Execution time: mean = 1.345 ms, total = 40.357 ms, Queueing time: mean = 76.850 us, max = 315.299 us, min = 30.033 us, total = 2.306 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 5 total (1 active, 1 running), Execution time: mean = 1.198 ms, total = 5.992 ms, Queueing time: mean = 46.862 us, max = 73.803 us, min = 35.406 us, total = 234.311 us NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:06:53,811 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 13738 total (1 active) Queueing time: mean = 51.718 us, max = 618.612 us, min = 2.750 us, total = 710.499 ms Execution time: mean = 264.420 us, total = 3.633 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 6237 total (0 active), Execution time: mean = 548.813 us, total = 3.423 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 6237 total (0 active), Execution time: mean = 28.188 us, total = 175.810 ms, Queueing time: mean = 111.165 us, max = 301.275 us, min = 5.787 us, total = 693.337 ms event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 27.529 us, total = 33.090 ms, Queueing time: mean = 9.984 us, max = 618.612 us, min = 2.750 us, total = 12.001 ms GcsTaskManager.GcJobSummary - 61 total (1 active), Execution time: mean = 12.291 us, total = 749.730 us, Queueing time: mean = 82.969 us, max = 159.415 us, min = 27.203 us, total = 5.061 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:06:53,811 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1591 total (25 active) Queueing time: mean = 23.417 us, max = 333.684 us, min = 2.330 us, total = 37.257 ms Execution time: mean = 185.104 ms, total = 294.500 s Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 29.216 us, total = 35.117 ms, Queueing time: mean = 10.907 us, max = 291.078 us, min = 2.330 us, total = 13.110 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 96 total (0 active), Execution time: mean = 43.276 us, total = 4.154 ms, Queueing time: mean = 124.648 us, max = 333.684 us, min = 14.442 us, total = 11.966 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 96 total (24 active), Execution time: mean = 3.067 s, total = 294.387 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 72 total (0 active), Execution time: mean = 119.161 us, total = 8.580 ms, Queueing time: mean = 120.984 us, max = 212.157 us, min = 31.977 us, total = 8.711 ms InternalPubSubGcsService.grpc_server.GcsPublish - 72 total (0 active), Execution time: mean = 675.075 us, total = 48.605 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.808 us, total = 15.616 us, Queueing time: mean = 26.258 us, max = 52.516 us, min = 52.516 us, total = 52.516 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:06:53,812 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1228 total (0 active) Queueing time: mean = 13.586 us, max = 1.423 ms, min = 568.000 ns, total = 16.684 ms Execution time: mean = 30.551 us, total = 37.516 ms Event stats: event_loop_lag_probe - 1202 total (0 active), Execution time: mean = 29.923 us, total = 35.967 ms, Queueing time: mean = 12.317 us, max = 1.423 ms, min = 1.850 us, total = 14.805 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:07:53,812 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 72 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 85 - cumulative published bytes: 62831 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:07:53,813 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 4747 total (4 active) Queueing time: mean = 577.256 us, max = 279.303 ms, min = 807.000 ns, total = 2.740 s Execution time: mean = 300.754 us, total = 1.428 s Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 25.877 us, total = 37.315 ms, Queueing time: mean = 19.400 us, max = 13.097 ms, min = 3.649 us, total = 27.974 ms GcsInMemoryStore.Put - 622 total (0 active), Execution time: mean = 456.709 us, total = 284.073 ms, Queueing time: mean = 3.183 ms, max = 278.545 ms, min = 2.310 us, total = 1.980 s RayletLoadPulled - 360 total (1 active), Execution time: mean = 235.656 us, total = 84.836 ms, Queueing time: mean = 75.185 us, max = 161.851 us, min = 19.183 us, total = 27.067 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 357 total (0 active), Execution time: mean = 76.326 us, total = 27.248 ms, Queueing time: mean = 109.148 us, max = 269.272 us, min = 18.998 us, total = 38.966 ms NodeManagerService.grpc_client.GetResourceLoad - 357 total (0 active), Execution time: mean = 1.476 ms, total = 526.966 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 158 total (0 active), Execution time: mean = 40.559 us, total = 6.408 ms, Queueing time: mean = 108.620 us, max = 193.151 us, min = 15.181 us, total = 17.162 ms InternalKVGcsService.grpc_server.InternalKVPut - 158 total (0 active), Execution time: mean = 587.161 us, total = 92.771 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 144 total (0 active), Execution time: mean = 31.414 us, total = 4.524 ms, Queueing time: mean = 121.736 us, max = 212.853 us, min = 26.210 us, total = 17.530 ms NodeInfoGcsService.grpc_server.CheckAlive - 144 total (0 active), Execution time: mean = 600.773 us, total = 86.511 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 141 total (0 active), Execution time: mean = 24.683 us, total = 3.480 ms, Queueing time: mean = 15.075 us, max = 64.163 us, min = 4.194 us, total = 2.126 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 140 total (0 active), Execution time: mean = 48.062 us, total = 6.729 ms, Queueing time: mean = 114.573 us, max = 269.544 us, min = 13.883 us, total = 16.040 ms InternalKVGcsService.grpc_server.InternalKVGet - 140 total (0 active), Execution time: mean = 622.512 us, total = 87.152 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 121 total (1 active), Execution time: mean = 10.523 us, total = 1.273 ms, Queueing time: mean = 82.699 us, max = 154.987 us, min = 29.516 us, total = 10.007 ms HealthCheck - 118 total (0 active), Execution time: mean = 8.292 us, total = 978.474 us, Queueing time: mean = 106.541 us, max = 219.229 us, min = 20.526 us, total = 12.572 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 72 total (0 active), Execution time: mean = 87.972 us, total = 6.334 ms, Queueing time: mean = 118.086 us, max = 190.124 us, min = 24.786 us, total = 8.502 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 72 total (0 active), Execution time: mean = 746.584 us, total = 53.754 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 36 total (1 active), Execution time: mean = 1.374 ms, total = 49.457 ms, Queueing time: mean = 79.822 us, max = 315.299 us, min = 30.033 us, total = 2.874 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us GCSServer.deadline_timer.debug_state_event_stats_print - 6 total (1 active, 1 running), Execution time: mean = 1.293 ms, total = 7.760 ms, Queueing time: mean = 49.895 us, max = 73.803 us, min = 35.406 us, total = 299.369 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:07:53,813 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 16510 total (1 active) Queueing time: mean = 51.353 us, max = 618.612 us, min = 2.750 us, total = 847.832 ms Execution time: mean = 263.544 us, total = 4.351 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 7497 total (0 active), Execution time: mean = 546.891 us, total = 4.100 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 7497 total (0 active), Execution time: mean = 28.123 us, total = 210.835 ms, Queueing time: mean = 110.372 us, max = 301.275 us, min = 5.787 us, total = 827.456 ms event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 27.274 us, total = 39.329 ms, Queueing time: mean = 9.820 us, max = 618.612 us, min = 2.750 us, total = 14.161 ms GcsTaskManager.GcJobSummary - 73 total (1 active), Execution time: mean = 12.383 us, total = 903.975 us, Queueing time: mean = 83.764 us, max = 159.415 us, min = 27.203 us, total = 6.115 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:07:53,813 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 1883 total (25 active) Queueing time: mean = 23.045 us, max = 333.684 us, min = 2.330 us, total = 43.394 ms Execution time: mean = 188.358 ms, total = 354.678 s Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 29.850 us, total = 43.043 ms, Queueing time: mean = 11.100 us, max = 291.078 us, min = 2.330 us, total = 16.006 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 109 total (0 active), Execution time: mean = 43.630 us, total = 4.756 ms, Queueing time: mean = 124.233 us, max = 333.684 us, min = 14.442 us, total = 13.541 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 109 total (24 active), Execution time: mean = 3.253 s, total = 354.546 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 85 total (0 active), Execution time: mean = 120.144 us, total = 10.212 ms, Queueing time: mean = 122.074 us, max = 247.252 us, min = 31.977 us, total = 10.376 ms InternalPubSubGcsService.grpc_server.GcsPublish - 85 total (0 active), Execution time: mean = 675.904 us, total = 57.452 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.808 us, total = 15.616 us, Queueing time: mean = 26.258 us, max = 52.516 us, min = 52.516 us, total = 52.516 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:07:53,813 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1468 total (0 active) Queueing time: mean = 13.303 us, max = 1.423 ms, min = 568.000 ns, total = 19.529 ms Execution time: mean = 30.763 us, total = 45.161 ms Event stats: event_loop_lag_probe - 1442 total (0 active), Execution time: mean = 30.244 us, total = 43.612 ms, Queueing time: mean = 12.240 us, max = 1.423 ms, min = 1.850 us, total = 17.650 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:08:53,814 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 84 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 97 - cumulative published bytes: 72647 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:08:53,815 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 5466 total (4 active) Queueing time: mean = 506.964 us, max = 279.303 ms, min = 807.000 ns, total = 2.771 s Execution time: mean = 293.122 us, total = 1.602 s Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 25.554 us, total = 42.982 ms, Queueing time: mean = 17.914 us, max = 13.097 ms, min = 3.184 us, total = 30.132 ms GcsInMemoryStore.Put - 718 total (0 active), Execution time: mean = 396.388 us, total = 284.606 ms, Queueing time: mean = 2.763 ms, max = 278.545 ms, min = 2.310 us, total = 1.984 s RayletLoadPulled - 420 total (1 active), Execution time: mean = 238.700 us, total = 100.254 ms, Queueing time: mean = 75.082 us, max = 161.851 us, min = 19.183 us, total = 31.534 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 417 total (0 active), Execution time: mean = 76.271 us, total = 31.805 ms, Queueing time: mean = 110.666 us, max = 269.272 us, min = 18.998 us, total = 46.148 ms NodeManagerService.grpc_client.GetResourceLoad - 417 total (0 active), Execution time: mean = 1.480 ms, total = 617.208 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 182 total (0 active), Execution time: mean = 40.248 us, total = 7.325 ms, Queueing time: mean = 108.754 us, max = 193.151 us, min = 15.181 us, total = 19.793 ms InternalKVGcsService.grpc_server.InternalKVPut - 182 total (0 active), Execution time: mean = 585.625 us, total = 106.584 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 168 total (0 active), Execution time: mean = 31.276 us, total = 5.254 ms, Queueing time: mean = 121.987 us, max = 212.853 us, min = 26.210 us, total = 20.494 ms NodeInfoGcsService.grpc_server.CheckAlive - 168 total (0 active), Execution time: mean = 597.502 us, total = 100.380 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 153 total (0 active), Execution time: mean = 24.246 us, total = 3.710 ms, Queueing time: mean = 14.929 us, max = 64.163 us, min = 4.194 us, total = 2.284 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 152 total (0 active), Execution time: mean = 48.640 us, total = 7.393 ms, Queueing time: mean = 114.862 us, max = 269.544 us, min = 13.883 us, total = 17.459 ms InternalKVGcsService.grpc_server.InternalKVGet - 152 total (0 active), Execution time: mean = 622.365 us, total = 94.599 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 141 total (1 active), Execution time: mean = 10.629 us, total = 1.499 ms, Queueing time: mean = 81.542 us, max = 154.987 us, min = 24.991 us, total = 11.497 ms HealthCheck - 138 total (0 active), Execution time: mean = 8.161 us, total = 1.126 ms, Queueing time: mean = 108.393 us, max = 399.856 us, min = 20.526 us, total = 14.958 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 84 total (0 active), Execution time: mean = 89.527 us, total = 7.520 ms, Queueing time: mean = 119.090 us, max = 195.939 us, min = 24.786 us, total = 10.004 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 84 total (0 active), Execution time: mean = 744.296 us, total = 62.521 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 42 total (1 active), Execution time: mean = 1.377 ms, total = 57.837 ms, Queueing time: mean = 77.051 us, max = 315.299 us, min = 25.112 us, total = 3.236 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_event_stats_print - 7 total (1 active, 1 running), Execution time: mean = 1.356 ms, total = 9.495 ms, Queueing time: mean = 52.372 us, max = 73.803 us, min = 35.406 us, total = 366.606 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:08:53,815 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 19282 total (1 active) Queueing time: mean = 52.009 us, max = 7.565 ms, min = 2.750 us, total = 1.003 s Execution time: mean = 264.456 us, total = 5.099 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 8757 total (0 active), Execution time: mean = 548.321 us, total = 4.802 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 8757 total (0 active), Execution time: mean = 28.625 us, total = 250.665 ms, Queueing time: mean = 111.832 us, max = 7.565 ms, min = 5.787 us, total = 979.310 ms event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 27.292 us, total = 45.904 ms, Queueing time: mean = 9.764 us, max = 618.612 us, min = 2.750 us, total = 16.423 ms GcsTaskManager.GcJobSummary - 85 total (1 active), Execution time: mean = 12.046 us, total = 1.024 ms, Queueing time: mean = 82.446 us, max = 159.415 us, min = 27.203 us, total = 7.008 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:08:53,815 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2171 total (25 active) Queueing time: mean = 22.490 us, max = 333.684 us, min = 2.330 us, total = 48.825 ms Execution time: mean = 191.082 ms, total = 414.839 s Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 29.952 us, total = 50.380 ms, Queueing time: mean = 11.141 us, max = 291.078 us, min = 2.330 us, total = 18.739 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 121 total (0 active), Execution time: mean = 44.268 us, total = 5.356 ms, Queueing time: mean = 122.744 us, max = 333.684 us, min = 14.442 us, total = 14.852 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 121 total (24 active), Execution time: mean = 3.427 s, total = 414.689 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 97 total (0 active), Execution time: mean = 121.421 us, total = 11.778 ms, Queueing time: mean = 121.276 us, max = 247.252 us, min = 31.977 us, total = 11.764 ms InternalPubSubGcsService.grpc_server.GcsPublish - 97 total (0 active), Execution time: mean = 678.561 us, total = 65.820 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.808 us, total = 15.616 us, Queueing time: mean = 26.258 us, max = 52.516 us, min = 52.516 us, total = 52.516 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:08:53,815 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1708 total (0 active) Queueing time: mean = 12.837 us, max = 1.423 ms, min = 568.000 ns, total = 21.926 ms Execution time: mean = 30.204 us, total = 51.588 ms Event stats: event_loop_lag_probe - 1682 total (0 active), Execution time: mean = 29.749 us, total = 50.039 ms, Queueing time: mean = 11.919 us, max = 1.423 ms, min = 1.850 us, total = 20.047 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:09:53,815 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 96 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 109 - cumulative published bytes: 82463 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:09:53,816 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 6185 total (4 active) Queueing time: mean = 452.708 us, max = 279.303 ms, min = 807.000 ns, total = 2.800 s Execution time: mean = 286.653 us, total = 1.773 s Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 25.461 us, total = 48.936 ms, Queueing time: mean = 16.754 us, max = 13.097 ms, min = 3.184 us, total = 32.202 ms GcsInMemoryStore.Put - 814 total (0 active), Execution time: mean = 350.307 us, total = 285.150 ms, Queueing time: mean = 2.442 ms, max = 278.545 ms, min = 2.310 us, total = 1.988 s RayletLoadPulled - 480 total (1 active), Execution time: mean = 238.312 us, total = 114.390 ms, Queueing time: mean = 75.156 us, max = 161.851 us, min = 19.183 us, total = 36.075 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 477 total (0 active), Execution time: mean = 75.962 us, total = 36.234 ms, Queueing time: mean = 110.414 us, max = 269.272 us, min = 18.799 us, total = 52.668 ms NodeManagerService.grpc_client.GetResourceLoad - 477 total (0 active), Execution time: mean = 1.478 ms, total = 704.919 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 206 total (0 active), Execution time: mean = 40.522 us, total = 8.348 ms, Queueing time: mean = 107.753 us, max = 193.151 us, min = 15.181 us, total = 22.197 ms InternalKVGcsService.grpc_server.InternalKVPut - 206 total (0 active), Execution time: mean = 584.999 us, total = 120.510 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 192 total (0 active), Execution time: mean = 30.969 us, total = 5.946 ms, Queueing time: mean = 120.361 us, max = 212.853 us, min = 26.210 us, total = 23.109 ms NodeInfoGcsService.grpc_server.CheckAlive - 192 total (0 active), Execution time: mean = 592.946 us, total = 113.846 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Get - 165 total (0 active), Execution time: mean = 24.130 us, total = 3.981 ms, Queueing time: mean = 14.859 us, max = 64.163 us, min = 4.194 us, total = 2.452 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 164 total (0 active), Execution time: mean = 49.832 us, total = 8.173 ms, Queueing time: mean = 114.217 us, max = 269.544 us, min = 13.883 us, total = 18.732 ms InternalKVGcsService.grpc_server.InternalKVGet - 164 total (0 active), Execution time: mean = 622.197 us, total = 102.040 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 161 total (1 active), Execution time: mean = 10.400 us, total = 1.674 ms, Queueing time: mean = 79.578 us, max = 154.987 us, min = 24.991 us, total = 12.812 ms HealthCheck - 158 total (0 active), Execution time: mean = 8.087 us, total = 1.278 ms, Queueing time: mean = 107.500 us, max = 399.856 us, min = 20.526 us, total = 16.985 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 96 total (0 active), Execution time: mean = 91.554 us, total = 8.789 ms, Queueing time: mean = 117.825 us, max = 195.939 us, min = 24.786 us, total = 11.311 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 96 total (0 active), Execution time: mean = 740.714 us, total = 71.109 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 48 total (1 active), Execution time: mean = 1.382 ms, total = 66.358 ms, Queueing time: mean = 74.417 us, max = 315.299 us, min = 18.624 us, total = 3.572 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_event_stats_print - 8 total (1 active, 1 running), Execution time: mean = 1.396 ms, total = 11.165 ms, Queueing time: mean = 55.546 us, max = 77.761 us, min = 35.406 us, total = 444.367 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:09:53,817 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 22054 total (1 active) Queueing time: mean = 51.698 us, max = 7.565 ms, min = 2.750 us, total = 1.140 s Execution time: mean = 264.169 us, total = 5.826 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 10017 total (0 active), Execution time: mean = 547.693 us, total = 5.486 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 10017 total (0 active), Execution time: mean = 28.561 us, total = 286.094 ms, Queueing time: mean = 111.132 us, max = 7.565 ms, min = 5.787 us, total = 1.113 s event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 27.305 us, total = 52.479 ms, Queueing time: mean = 9.767 us, max = 618.612 us, min = 2.750 us, total = 18.772 ms GcsTaskManager.GcJobSummary - 97 total (1 active), Execution time: mean = 11.896 us, total = 1.154 ms, Queueing time: mean = 83.012 us, max = 171.691 us, min = 27.203 us, total = 8.052 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:09:53,817 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2459 total (25 active) Queueing time: mean = 22.026 us, max = 333.684 us, min = 2.330 us, total = 54.161 ms Execution time: mean = 193.171 ms, total = 475.009 s Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 30.067 us, total = 57.789 ms, Queueing time: mean = 11.215 us, max = 291.078 us, min = 2.330 us, total = 21.555 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 133 total (0 active), Execution time: mean = 45.349 us, total = 6.031 ms, Queueing time: mean = 120.589 us, max = 333.684 us, min = 14.442 us, total = 16.038 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 133 total (24 active), Execution time: mean = 3.570 s, total = 474.841 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 109 total (0 active), Execution time: mean = 121.882 us, total = 13.285 ms, Queueing time: mean = 120.159 us, max = 247.252 us, min = 27.743 us, total = 13.097 ms InternalPubSubGcsService.grpc_server.GcsPublish - 109 total (0 active), Execution time: mean = 677.176 us, total = 73.812 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.808 us, total = 15.616 us, Queueing time: mean = 26.258 us, max = 52.516 us, min = 52.516 us, total = 52.516 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:09:53,817 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 1948 total (0 active) Queueing time: mean = 12.420 us, max = 1.423 ms, min = 568.000 ns, total = 24.194 ms Execution time: mean = 29.718 us, total = 57.890 ms Event stats: event_loop_lag_probe - 1922 total (0 active), Execution time: mean = 29.314 us, total = 56.341 ms, Queueing time: mean = 11.610 us, max = 1.423 ms, min = 1.850 us, total = 22.314 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:10:53,817 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 108 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 121 - cumulative published bytes: 92279 - current buffered bytes: 818 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:10:53,818 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 6904 total (4 active) Queueing time: mean = 409.202 us, max = 279.303 ms, min = 807.000 ns, total = 2.825 s Execution time: mean = 280.400 us, total = 1.936 s Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 24.997 us, total = 54.044 ms, Queueing time: mean = 15.827 us, max = 13.097 ms, min = 2.755 us, total = 34.218 ms GcsInMemoryStore.Put - 910 total (0 active), Execution time: mean = 313.841 us, total = 285.595 ms, Queueing time: mean = 2.189 ms, max = 278.545 ms, min = 2.310 us, total = 1.992 s RayletLoadPulled - 540 total (1 active), Execution time: mean = 235.051 us, total = 126.927 ms, Queueing time: mean = 74.190 us, max = 164.256 us, min = 13.284 us, total = 40.063 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 537 total (0 active), Execution time: mean = 74.833 us, total = 40.185 ms, Queueing time: mean = 107.600 us, max = 269.272 us, min = 12.992 us, total = 57.781 ms NodeManagerService.grpc_client.GetResourceLoad - 537 total (0 active), Execution time: mean = 1.475 ms, total = 791.920 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 230 total (0 active), Execution time: mean = 40.156 us, total = 9.236 ms, Queueing time: mean = 103.265 us, max = 193.151 us, min = 11.819 us, total = 23.751 ms InternalKVGcsService.grpc_server.InternalKVPut - 230 total (0 active), Execution time: mean = 575.569 us, total = 132.381 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 216 total (0 active), Execution time: mean = 30.680 us, total = 6.627 ms, Queueing time: mean = 119.910 us, max = 233.188 us, min = 20.329 us, total = 25.901 ms NodeInfoGcsService.grpc_server.CheckAlive - 216 total (0 active), Execution time: mean = 590.272 us, total = 127.499 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 181 total (1 active), Execution time: mean = 10.179 us, total = 1.842 ms, Queueing time: mean = 78.001 us, max = 154.987 us, min = 19.812 us, total = 14.118 ms HealthCheck - 178 total (0 active), Execution time: mean = 7.955 us, total = 1.416 ms, Queueing time: mean = 106.185 us, max = 399.856 us, min = 19.919 us, total = 18.901 ms GcsInMemoryStore.Get - 177 total (0 active), Execution time: mean = 24.470 us, total = 4.331 ms, Queueing time: mean = 14.628 us, max = 64.163 us, min = 4.194 us, total = 2.589 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 176 total (0 active), Execution time: mean = 49.683 us, total = 8.744 ms, Queueing time: mean = 111.990 us, max = 269.544 us, min = 13.883 us, total = 19.710 ms InternalKVGcsService.grpc_server.InternalKVGet - 176 total (0 active), Execution time: mean = 618.608 us, total = 108.875 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 108 total (0 active), Execution time: mean = 90.897 us, total = 9.817 ms, Queueing time: mean = 115.874 us, max = 209.197 us, min = 24.786 us, total = 12.514 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 108 total (0 active), Execution time: mean = 727.116 us, total = 78.529 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 54 total (1 active), Execution time: mean = 1.386 ms, total = 74.840 ms, Queueing time: mean = 73.717 us, max = 315.299 us, min = 18.624 us, total = 3.981 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_event_stats_print - 9 total (1 active, 1 running), Execution time: mean = 1.441 ms, total = 12.966 ms, Queueing time: mean = 52.406 us, max = 77.761 us, min = 27.290 us, total = 471.657 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:10:53,818 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 24826 total (1 active) Queueing time: mean = 51.294 us, max = 7.565 ms, min = 2.750 us, total = 1.273 s Execution time: mean = 262.252 us, total = 6.511 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 11277 total (0 active), Execution time: mean = 543.982 us, total = 6.134 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 11277 total (0 active), Execution time: mean = 28.066 us, total = 316.500 ms, Queueing time: mean = 110.275 us, max = 7.565 ms, min = 5.787 us, total = 1.244 s event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 27.011 us, total = 58.399 ms, Queueing time: mean = 9.678 us, max = 618.612 us, min = 2.750 us, total = 20.924 ms GcsTaskManager.GcJobSummary - 109 total (1 active), Execution time: mean = 11.600 us, total = 1.264 ms, Queueing time: mean = 80.904 us, max = 171.691 us, min = 27.203 us, total = 8.819 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:10:53,818 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 2747 total (25 active) Queueing time: mean = 21.418 us, max = 333.684 us, min = 2.330 us, total = 58.834 ms Execution time: mean = 194.821 ms, total = 535.173 s Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 30.097 us, total = 65.069 ms, Queueing time: mean = 11.244 us, max = 291.078 us, min = 2.330 us, total = 24.310 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 145 total (0 active), Execution time: mean = 45.328 us, total = 6.573 ms, Queueing time: mean = 117.018 us, max = 333.684 us, min = 14.442 us, total = 16.968 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 145 total (24 active), Execution time: mean = 3.690 s, total = 534.990 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 121 total (0 active), Execution time: mean = 118.255 us, total = 14.309 ms, Queueing time: mean = 116.417 us, max = 247.252 us, min = 23.656 us, total = 14.086 ms InternalPubSubGcsService.grpc_server.GcsPublish - 121 total (0 active), Execution time: mean = 666.419 us, total = 80.637 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 2 total (1 active), Execution time: mean = 7.808 us, total = 15.616 us, Queueing time: mean = 26.258 us, max = 52.516 us, min = 52.516 us, total = 52.516 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:10:53,819 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2188 total (0 active) Queueing time: mean = 12.033 us, max = 1.423 ms, min = 568.000 ns, total = 26.329 ms Execution time: mean = 29.239 us, total = 63.975 ms Event stats: event_loop_lag_probe - 2162 total (0 active), Execution time: mean = 28.874 us, total = 62.426 ms, Queueing time: mean = 11.309 us, max = 1.423 ms, min = 1.850 us, total = 24.450 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:11:53,819 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 120 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 134 - cumulative published bytes: 102137 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:11:53,820 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 7622 total (4 active) Queueing time: mean = 374.200 us, max = 279.303 ms, min = 807.000 ns, total = 2.852 s Execution time: mean = 274.765 us, total = 2.094 s Event stats: event_loop_lag_probe - 2401 total (0 active), Execution time: mean = 24.763 us, total = 59.456 ms, Queueing time: mean = 15.133 us, max = 13.097 ms, min = 2.088 us, total = 36.335 ms GcsInMemoryStore.Put - 1006 total (0 active), Execution time: mean = 284.365 us, total = 286.071 ms, Queueing time: mean = 1.984 ms, max = 278.545 ms, min = 2.310 us, total = 1.996 s RayletLoadPulled - 600 total (1 active), Execution time: mean = 233.964 us, total = 140.378 ms, Queueing time: mean = 74.547 us, max = 266.426 us, min = 13.284 us, total = 44.728 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 597 total (0 active), Execution time: mean = 74.434 us, total = 44.437 ms, Queueing time: mean = 105.325 us, max = 269.272 us, min = 12.992 us, total = 62.879 ms NodeManagerService.grpc_client.GetResourceLoad - 597 total (0 active), Execution time: mean = 1.462 ms, total = 872.637 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 254 total (0 active), Execution time: mean = 39.594 us, total = 10.057 ms, Queueing time: mean = 102.068 us, max = 193.151 us, min = 11.443 us, total = 25.925 ms InternalKVGcsService.grpc_server.InternalKVPut - 254 total (0 active), Execution time: mean = 568.705 us, total = 144.451 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 240 total (0 active), Execution time: mean = 30.578 us, total = 7.339 ms, Queueing time: mean = 118.542 us, max = 233.188 us, min = 20.329 us, total = 28.450 ms NodeInfoGcsService.grpc_server.CheckAlive - 240 total (0 active), Execution time: mean = 586.174 us, total = 140.682 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 201 total (1 active), Execution time: mean = 10.015 us, total = 2.013 ms, Queueing time: mean = 76.541 us, max = 154.987 us, min = 19.812 us, total = 15.385 ms HealthCheck - 198 total (0 active), Execution time: mean = 7.871 us, total = 1.558 ms, Queueing time: mean = 105.874 us, max = 399.856 us, min = 19.919 us, total = 20.963 ms GcsInMemoryStore.Get - 189 total (0 active), Execution time: mean = 25.711 us, total = 4.859 ms, Queueing time: mean = 14.535 us, max = 64.163 us, min = 4.194 us, total = 2.747 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 188 total (0 active), Execution time: mean = 49.671 us, total = 9.338 ms, Queueing time: mean = 110.650 us, max = 269.544 us, min = 13.883 us, total = 20.802 ms InternalKVGcsService.grpc_server.InternalKVGet - 188 total (0 active), Execution time: mean = 614.897 us, total = 115.601 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 120 total (0 active), Execution time: mean = 90.175 us, total = 10.821 ms, Queueing time: mean = 113.736 us, max = 209.197 us, min = 22.231 us, total = 13.648 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 120 total (0 active), Execution time: mean = 712.536 us, total = 85.504 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 60 total (1 active), Execution time: mean = 1.406 ms, total = 84.363 ms, Queueing time: mean = 73.643 us, max = 315.299 us, min = 18.624 us, total = 4.419 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_event_stats_print - 10 total (1 active, 1 running), Execution time: mean = 1.459 ms, total = 14.586 ms, Queueing time: mean = 55.020 us, max = 78.539 us, min = 27.290 us, total = 550.196 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:11:53,820 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 27598 total (1 active) Queueing time: mean = 50.869 us, max = 7.565 ms, min = 2.750 us, total = 1.404 s Execution time: mean = 259.870 us, total = 7.172 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 12537 total (0 active), Execution time: mean = 539.001 us, total = 6.757 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 12537 total (0 active), Execution time: mean = 27.799 us, total = 348.513 ms, Queueing time: mean = 109.363 us, max = 7.565 ms, min = 5.787 us, total = 1.371 s event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 26.860 us, total = 64.517 ms, Queueing time: mean = 9.591 us, max = 618.612 us, min = 2.750 us, total = 23.038 ms GcsTaskManager.GcJobSummary - 121 total (1 active), Execution time: mean = 11.562 us, total = 1.399 ms, Queueing time: mean = 79.859 us, max = 171.691 us, min = 27.203 us, total = 9.663 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:11:53,820 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3086 total (25 active) Queueing time: mean = 21.638 us, max = 333.684 us, min = 2.330 us, total = 66.776 ms Execution time: mean = 4.640 s, total = 14319.442 s Event stats: event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 30.079 us, total = 72.249 ms, Queueing time: mean = 11.225 us, max = 291.078 us, min = 2.330 us, total = 26.962 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 181 total (0 active), Execution time: mean = 42.627 us, total = 7.715 ms, Queueing time: mean = 114.738 us, max = 333.684 us, min = 14.442 us, total = 20.768 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 181 total (24 active), Execution time: mean = 79.112 s, total = 14319.238 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 134 total (0 active), Execution time: mean = 117.706 us, total = 15.773 ms, Queueing time: mean = 115.782 us, max = 247.252 us, min = 23.656 us, total = 15.515 ms InternalPubSubGcsService.grpc_server.GcsPublish - 134 total (0 active), Execution time: mean = 665.900 us, total = 89.231 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 600.277 us, total = 1.801 ms, Queueing time: mean = 38.188 us, max = 62.049 us, min = 52.516 us, total = 114.565 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:11:53,820 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2428 total (0 active) Queueing time: mean = 11.699 us, max = 1.423 ms, min = 568.000 ns, total = 28.404 ms Execution time: mean = 28.745 us, total = 69.793 ms Event stats: event_loop_lag_probe - 2402 total (0 active), Execution time: mean = 28.411 us, total = 68.244 ms, Queueing time: mean = 11.043 us, max = 1.423 ms, min = 1.850 us, total = 26.525 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:12:53,820 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 132 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 147 - cumulative published bytes: 111999 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:12:53,821 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 8341 total (4 active) Queueing time: mean = 344.903 us, max = 279.303 ms, min = 807.000 ns, total = 2.877 s Execution time: mean = 269.390 us, total = 2.247 s Event stats: event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 24.501 us, total = 64.706 ms, Queueing time: mean = 14.536 us, max = 13.097 ms, min = 2.088 us, total = 38.389 ms GcsInMemoryStore.Put - 1102 total (0 active), Execution time: mean = 260.030 us, total = 286.553 ms, Queueing time: mean = 1.815 ms, max = 278.545 ms, min = 2.310 us, total = 2.000 s RayletLoadPulled - 660 total (1 active), Execution time: mean = 231.577 us, total = 152.841 ms, Queueing time: mean = 73.397 us, max = 266.426 us, min = 11.570 us, total = 48.442 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 657 total (0 active), Execution time: mean = 73.787 us, total = 48.478 ms, Queueing time: mean = 103.109 us, max = 269.272 us, min = 12.278 us, total = 67.743 ms NodeManagerService.grpc_client.GetResourceLoad - 657 total (0 active), Execution time: mean = 1.451 ms, total = 953.223 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 278 total (0 active), Execution time: mean = 39.584 us, total = 11.004 ms, Queueing time: mean = 98.744 us, max = 193.151 us, min = 11.443 us, total = 27.451 ms InternalKVGcsService.grpc_server.InternalKVPut - 278 total (0 active), Execution time: mean = 561.786 us, total = 156.177 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 264 total (0 active), Execution time: mean = 30.704 us, total = 8.106 ms, Queueing time: mean = 118.448 us, max = 233.188 us, min = 20.329 us, total = 31.270 ms NodeInfoGcsService.grpc_server.CheckAlive - 264 total (0 active), Execution time: mean = 582.119 us, total = 153.679 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 221 total (1 active), Execution time: mean = 9.949 us, total = 2.199 ms, Queueing time: mean = 75.326 us, max = 154.987 us, min = 19.812 us, total = 16.647 ms HealthCheck - 218 total (0 active), Execution time: mean = 7.807 us, total = 1.702 ms, Queueing time: mean = 104.010 us, max = 399.856 us, min = 19.476 us, total = 22.674 ms GcsInMemoryStore.Get - 201 total (0 active), Execution time: mean = 25.271 us, total = 5.079 ms, Queueing time: mean = 14.305 us, max = 64.163 us, min = 4.068 us, total = 2.875 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 200 total (0 active), Execution time: mean = 49.377 us, total = 9.875 ms, Queueing time: mean = 108.742 us, max = 269.544 us, min = 13.883 us, total = 21.748 ms InternalKVGcsService.grpc_server.InternalKVGet - 200 total (0 active), Execution time: mean = 608.792 us, total = 121.758 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 132 total (0 active), Execution time: mean = 89.022 us, total = 11.751 ms, Queueing time: mean = 112.102 us, max = 209.197 us, min = 19.828 us, total = 14.797 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 132 total (0 active), Execution time: mean = 699.792 us, total = 92.373 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 66 total (1 active), Execution time: mean = 1.384 ms, total = 91.312 ms, Queueing time: mean = 72.632 us, max = 315.299 us, min = 17.370 us, total = 4.794 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GCSServer.deadline_timer.debug_state_event_stats_print - 11 total (1 active, 1 running), Execution time: mean = 1.460 ms, total = 16.056 ms, Queueing time: mean = 55.179 us, max = 78.539 us, min = 27.290 us, total = 606.967 us GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:12:53,821 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 30369 total (1 active) Queueing time: mean = 50.497 us, max = 7.565 ms, min = 2.750 us, total = 1.534 s Execution time: mean = 258.229 us, total = 7.842 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 13797 total (0 active), Execution time: mean = 535.652 us, total = 7.390 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 13797 total (0 active), Execution time: mean = 27.525 us, total = 379.758 ms, Queueing time: mean = 108.556 us, max = 7.565 ms, min = 5.787 us, total = 1.498 s event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 26.680 us, total = 70.461 ms, Queueing time: mean = 9.540 us, max = 618.612 us, min = 2.750 us, total = 25.194 ms GcsTaskManager.GcJobSummary - 133 total (1 active), Execution time: mean = 11.551 us, total = 1.536 ms, Queueing time: mean = 79.002 us, max = 171.691 us, min = 27.203 us, total = 10.507 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:12:53,822 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3377 total (25 active) Queueing time: mean = 21.239 us, max = 333.684 us, min = 2.330 us, total = 71.724 ms Execution time: mean = 4.258 s, total = 14379.585 s Event stats: event_loop_lag_probe - 2641 total (0 active), Execution time: mean = 30.013 us, total = 79.265 ms, Queueing time: mean = 11.151 us, max = 291.078 us, min = 2.330 us, total = 29.449 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 194 total (0 active), Execution time: mean = 43.217 us, total = 8.384 ms, Queueing time: mean = 112.793 us, max = 333.684 us, min = 14.442 us, total = 21.882 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 194 total (24 active), Execution time: mean = 74.120 s, total = 14379.364 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 147 total (0 active), Execution time: mean = 117.661 us, total = 17.296 ms, Queueing time: mean = 114.700 us, max = 247.252 us, min = 23.656 us, total = 16.861 ms InternalPubSubGcsService.grpc_server.GcsPublish - 147 total (0 active), Execution time: mean = 662.923 us, total = 97.450 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 600.277 us, total = 1.801 ms, Queueing time: mean = 38.188 us, max = 62.049 us, min = 52.516 us, total = 114.565 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:12:53,822 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2668 total (0 active) Queueing time: mean = 11.536 us, max = 1.423 ms, min = 568.000 ns, total = 30.779 ms Execution time: mean = 28.533 us, total = 76.127 ms Event stats: event_loop_lag_probe - 2642 total (0 active), Execution time: mean = 28.228 us, total = 74.578 ms, Queueing time: mean = 10.939 us, max = 1.423 ms, min = 1.850 us, total = 28.900 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:13:53,822 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 144 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 160 - cumulative published bytes: 121861 - current buffered bytes: 153 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:13:53,823 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 9060 total (4 active) Queueing time: mean = 321.130 us, max = 279.303 ms, min = 807.000 ns, total = 2.909 s Execution time: mean = 268.084 us, total = 2.429 s Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 24.385 us, total = 70.253 ms, Queueing time: mean = 14.059 us, max = 13.097 ms, min = 2.088 us, total = 40.504 ms GcsInMemoryStore.Put - 1198 total (0 active), Execution time: mean = 239.630 us, total = 287.077 ms, Queueing time: mean = 1.673 ms, max = 278.545 ms, min = 2.310 us, total = 2.005 s RayletLoadPulled - 720 total (1 active), Execution time: mean = 233.169 us, total = 167.881 ms, Queueing time: mean = 74.243 us, max = 266.426 us, min = 11.570 us, total = 53.455 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 717 total (0 active), Execution time: mean = 74.399 us, total = 53.344 ms, Queueing time: mean = 104.349 us, max = 269.272 us, min = 12.278 us, total = 74.818 ms NodeManagerService.grpc_client.GetResourceLoad - 717 total (0 active), Execution time: mean = 1.461 ms, total = 1.048 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 302 total (0 active), Execution time: mean = 40.589 us, total = 12.258 ms, Queueing time: mean = 100.138 us, max = 193.151 us, min = 11.443 us, total = 30.242 ms InternalKVGcsService.grpc_server.InternalKVPut - 302 total (0 active), Execution time: mean = 565.745 us, total = 170.855 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 288 total (0 active), Execution time: mean = 31.904 us, total = 9.188 ms, Queueing time: mean = 119.535 us, max = 233.188 us, min = 20.329 us, total = 34.426 ms NodeInfoGcsService.grpc_server.CheckAlive - 288 total (0 active), Execution time: mean = 586.652 us, total = 168.956 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 241 total (1 active), Execution time: mean = 10.020 us, total = 2.415 ms, Queueing time: mean = 76.142 us, max = 154.987 us, min = 19.812 us, total = 18.350 ms HealthCheck - 238 total (0 active), Execution time: mean = 7.811 us, total = 1.859 ms, Queueing time: mean = 105.187 us, max = 399.856 us, min = 19.476 us, total = 25.034 ms GcsInMemoryStore.Get - 213 total (0 active), Execution time: mean = 25.052 us, total = 5.336 ms, Queueing time: mean = 14.342 us, max = 64.163 us, min = 4.068 us, total = 3.055 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 212 total (0 active), Execution time: mean = 50.243 us, total = 10.651 ms, Queueing time: mean = 109.286 us, max = 269.544 us, min = 13.883 us, total = 23.169 ms InternalKVGcsService.grpc_server.InternalKVGet - 212 total (0 active), Execution time: mean = 610.384 us, total = 129.401 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 144 total (0 active), Execution time: mean = 90.201 us, total = 12.989 ms, Queueing time: mean = 114.434 us, max = 296.469 us, min = 19.828 us, total = 16.479 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 144 total (0 active), Execution time: mean = 703.967 us, total = 101.371 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 72 total (1 active), Execution time: mean = 1.385 ms, total = 99.724 ms, Queueing time: mean = 72.966 us, max = 315.299 us, min = 17.370 us, total = 5.254 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 12 total (1 active, 1 running), Execution time: mean = 1.472 ms, total = 17.660 ms, Queueing time: mean = 55.817 us, max = 78.539 us, min = 27.290 us, total = 669.807 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:13:53,823 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 33141 total (1 active) Queueing time: mean = 50.570 us, max = 7.565 ms, min = 2.750 us, total = 1.676 s Execution time: mean = 258.844 us, total = 8.578 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 15057 total (0 active), Execution time: mean = 536.681 us, total = 8.081 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 15057 total (0 active), Execution time: mean = 27.758 us, total = 417.945 ms, Queueing time: mean = 108.704 us, max = 7.565 ms, min = 5.787 us, total = 1.637 s event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 27.013 us, total = 77.825 ms, Queueing time: mean = 9.594 us, max = 618.612 us, min = 2.750 us, total = 27.641 ms GcsTaskManager.GcJobSummary - 145 total (1 active), Execution time: mean = 12.150 us, total = 1.762 ms, Queueing time: mean = 78.894 us, max = 171.691 us, min = 27.203 us, total = 11.440 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:13:53,823 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3669 total (25 active) Queueing time: mean = 21.038 us, max = 333.684 us, min = 2.330 us, total = 77.187 ms Execution time: mean = 3.936 s, total = 14439.689 s Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 30.234 us, total = 87.103 ms, Queueing time: mean = 11.211 us, max = 291.078 us, min = 2.330 us, total = 32.299 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 207 total (0 active), Execution time: mean = 43.111 us, total = 8.924 ms, Queueing time: mean = 111.613 us, max = 333.684 us, min = 14.442 us, total = 23.104 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 207 total (24 active), Execution time: mean = 69.756 s, total = 14439.450 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 160 total (0 active), Execution time: mean = 117.581 us, total = 18.813 ms, Queueing time: mean = 114.079 us, max = 247.252 us, min = 23.656 us, total = 18.253 ms InternalPubSubGcsService.grpc_server.GcsPublish - 160 total (0 active), Execution time: mean = 664.874 us, total = 106.380 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 600.277 us, total = 1.801 ms, Queueing time: mean = 38.188 us, max = 62.049 us, min = 52.516 us, total = 114.565 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:13:53,823 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 2907 total (0 active) Queueing time: mean = 11.431 us, max = 1.423 ms, min = 568.000 ns, total = 33.230 ms Execution time: mean = 28.489 us, total = 82.816 ms Event stats: event_loop_lag_probe - 2881 total (0 active), Execution time: mean = 28.208 us, total = 81.267 ms, Queueing time: mean = 10.882 us, max = 1.423 ms, min = 1.850 us, total = 31.351 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:14:53,823 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 156 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 173 - cumulative published bytes: 131723 - current buffered bytes: 153 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:14:53,824 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 9779 total (4 active) Queueing time: mean = 300.786 us, max = 279.303 ms, min = 807.000 ns, total = 2.941 s Execution time: mean = 266.918 us, total = 2.610 s Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 24.358 us, total = 76.020 ms, Queueing time: mean = 13.691 us, max = 13.097 ms, min = 2.088 us, total = 42.729 ms GcsInMemoryStore.Put - 1294 total (0 active), Execution time: mean = 222.255 us, total = 287.598 ms, Queueing time: mean = 1.553 ms, max = 278.545 ms, min = 2.310 us, total = 2.009 s RayletLoadPulled - 780 total (1 active), Execution time: mean = 235.678 us, total = 183.829 ms, Queueing time: mean = 74.241 us, max = 266.426 us, min = 11.570 us, total = 57.908 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 777 total (0 active), Execution time: mean = 74.811 us, total = 58.128 ms, Queueing time: mean = 104.840 us, max = 269.272 us, min = 12.278 us, total = 81.460 ms NodeManagerService.grpc_client.GetResourceLoad - 777 total (0 active), Execution time: mean = 1.469 ms, total = 1.141 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 326 total (0 active), Execution time: mean = 40.857 us, total = 13.319 ms, Queueing time: mean = 102.341 us, max = 200.615 us, min = 11.443 us, total = 33.363 ms InternalKVGcsService.grpc_server.InternalKVPut - 326 total (0 active), Execution time: mean = 569.145 us, total = 185.541 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 312 total (0 active), Execution time: mean = 32.359 us, total = 10.096 ms, Queueing time: mean = 119.671 us, max = 233.188 us, min = 20.329 us, total = 37.337 ms NodeInfoGcsService.grpc_server.CheckAlive - 312 total (0 active), Execution time: mean = 589.761 us, total = 184.005 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 261 total (1 active), Execution time: mean = 10.067 us, total = 2.628 ms, Queueing time: mean = 77.151 us, max = 232.777 us, min = 19.812 us, total = 20.136 ms HealthCheck - 258 total (0 active), Execution time: mean = 7.815 us, total = 2.016 ms, Queueing time: mean = 105.112 us, max = 399.856 us, min = 19.476 us, total = 27.119 ms GcsInMemoryStore.Get - 225 total (0 active), Execution time: mean = 24.790 us, total = 5.578 ms, Queueing time: mean = 14.302 us, max = 64.163 us, min = 4.068 us, total = 3.218 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 224 total (0 active), Execution time: mean = 50.926 us, total = 11.407 ms, Queueing time: mean = 110.955 us, max = 269.544 us, min = 13.883 us, total = 24.854 ms InternalKVGcsService.grpc_server.InternalKVGet - 224 total (0 active), Execution time: mean = 613.768 us, total = 137.484 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 156 total (0 active), Execution time: mean = 90.902 us, total = 14.181 ms, Queueing time: mean = 116.807 us, max = 296.469 us, min = 19.828 us, total = 18.222 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 156 total (0 active), Execution time: mean = 708.046 us, total = 110.455 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 78 total (1 active), Execution time: mean = 1.379 ms, total = 107.547 ms, Queueing time: mean = 72.668 us, max = 315.299 us, min = 17.370 us, total = 5.668 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 13 total (1 active, 1 running), Execution time: mean = 1.454 ms, total = 18.900 ms, Queueing time: mean = 55.274 us, max = 78.539 us, min = 27.290 us, total = 718.564 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:14:53,825 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 35913 total (1 active) Queueing time: mean = 50.469 us, max = 7.565 ms, min = 2.750 us, total = 1.812 s Execution time: mean = 259.093 us, total = 9.305 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 16317 total (0 active), Execution time: mean = 537.098 us, total = 8.764 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 16317 total (0 active), Execution time: mean = 27.843 us, total = 454.306 ms, Queueing time: mean = 108.467 us, max = 7.565 ms, min = 5.787 us, total = 1.770 s event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 27.162 us, total = 84.774 ms, Queueing time: mean = 9.614 us, max = 618.612 us, min = 2.750 us, total = 30.005 ms GcsTaskManager.GcJobSummary - 157 total (1 active), Execution time: mean = 12.152 us, total = 1.908 ms, Queueing time: mean = 79.742 us, max = 171.691 us, min = 27.203 us, total = 12.519 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:14:53,825 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 3961 total (25 active) Queueing time: mean = 20.943 us, max = 333.684 us, min = 2.330 us, total = 82.954 ms Execution time: mean = 3.661 s, total = 14499.887 s Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 30.442 us, total = 95.011 ms, Queueing time: mean = 11.257 us, max = 291.078 us, min = 2.330 us, total = 35.134 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 220 total (0 active), Execution time: mean = 44.127 us, total = 9.708 ms, Queueing time: mean = 111.591 us, max = 333.684 us, min = 14.442 us, total = 24.550 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 220 total (24 active), Execution time: mean = 65.907 s, total = 14499.627 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 173 total (0 active), Execution time: mean = 119.056 us, total = 20.597 ms, Queueing time: mean = 114.091 us, max = 247.252 us, min = 23.656 us, total = 19.738 ms InternalPubSubGcsService.grpc_server.GcsPublish - 173 total (0 active), Execution time: mean = 667.289 us, total = 115.441 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 600.277 us, total = 1.801 ms, Queueing time: mean = 38.188 us, max = 62.049 us, min = 52.516 us, total = 114.565 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:14:53,825 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3147 total (0 active) Queueing time: mean = 11.311 us, max = 1.423 ms, min = 568.000 ns, total = 35.596 ms Execution time: mean = 28.391 us, total = 89.345 ms Event stats: event_loop_lag_probe - 3121 total (0 active), Execution time: mean = 28.131 us, total = 87.796 ms, Queueing time: mean = 10.803 us, max = 1.423 ms, min = 1.850 us, total = 33.717 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:15:53,825 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 168 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 185 - cumulative published bytes: 141551 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:15:53,826 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 10498 total (4 active) Queueing time: mean = 282.980 us, max = 279.303 ms, min = 807.000 ns, total = 2.971 s Execution time: mean = 265.159 us, total = 2.784 s Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 24.260 us, total = 81.538 ms, Queueing time: mean = 13.359 us, max = 13.097 ms, min = 2.088 us, total = 44.898 ms GcsInMemoryStore.Put - 1390 total (0 active), Execution time: mean = 207.223 us, total = 288.039 ms, Queueing time: mean = 1.449 ms, max = 278.545 ms, min = 2.310 us, total = 2.014 s RayletLoadPulled - 840 total (1 active), Execution time: mean = 236.340 us, total = 198.525 ms, Queueing time: mean = 74.326 us, max = 266.426 us, min = 11.570 us, total = 62.433 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 837 total (0 active), Execution time: mean = 74.862 us, total = 62.660 ms, Queueing time: mean = 105.008 us, max = 269.272 us, min = 12.278 us, total = 87.891 ms NodeManagerService.grpc_client.GetResourceLoad - 837 total (0 active), Execution time: mean = 1.472 ms, total = 1.232 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 350 total (0 active), Execution time: mean = 40.763 us, total = 14.267 ms, Queueing time: mean = 101.502 us, max = 200.615 us, min = 11.443 us, total = 35.526 ms InternalKVGcsService.grpc_server.InternalKVPut - 350 total (0 active), Execution time: mean = 568.799 us, total = 199.080 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 336 total (0 active), Execution time: mean = 32.691 us, total = 10.984 ms, Queueing time: mean = 118.874 us, max = 233.188 us, min = 20.329 us, total = 39.942 ms NodeInfoGcsService.grpc_server.CheckAlive - 336 total (0 active), Execution time: mean = 588.214 us, total = 197.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 281 total (1 active), Execution time: mean = 10.063 us, total = 2.828 ms, Queueing time: mean = 76.488 us, max = 232.777 us, min = 19.812 us, total = 21.493 ms HealthCheck - 278 total (0 active), Execution time: mean = 7.763 us, total = 2.158 ms, Queueing time: mean = 104.690 us, max = 399.856 us, min = 19.476 us, total = 29.104 ms GcsInMemoryStore.Get - 237 total (0 active), Execution time: mean = 24.448 us, total = 5.794 ms, Queueing time: mean = 14.179 us, max = 64.163 us, min = 4.068 us, total = 3.360 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 236 total (0 active), Execution time: mean = 51.745 us, total = 12.212 ms, Queueing time: mean = 110.500 us, max = 269.544 us, min = 13.883 us, total = 26.078 ms InternalKVGcsService.grpc_server.InternalKVGet - 236 total (0 active), Execution time: mean = 613.423 us, total = 144.768 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 168 total (0 active), Execution time: mean = 91.081 us, total = 15.302 ms, Queueing time: mean = 115.750 us, max = 296.469 us, min = 19.828 us, total = 19.446 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 168 total (0 active), Execution time: mean = 708.350 us, total = 119.003 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 84 total (1 active), Execution time: mean = 1.380 ms, total = 115.946 ms, Queueing time: mean = 71.732 us, max = 315.299 us, min = 17.370 us, total = 6.026 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 14 total (1 active, 1 running), Execution time: mean = 1.469 ms, total = 20.565 ms, Queueing time: mean = 55.980 us, max = 78.539 us, min = 27.290 us, total = 783.727 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:15:53,826 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 38685 total (1 active) Queueing time: mean = 50.499 us, max = 7.565 ms, min = 2.216 us, total = 1.954 s Execution time: mean = 259.298 us, total = 10.031 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 17577 total (0 active), Execution time: mean = 537.353 us, total = 9.445 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 17577 total (0 active), Execution time: mean = 27.868 us, total = 489.840 ms, Queueing time: mean = 108.518 us, max = 7.565 ms, min = 5.787 us, total = 1.907 s event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 27.969 us, total = 94.004 ms, Queueing time: mean = 9.684 us, max = 618.612 us, min = 2.216 us, total = 32.549 ms GcsTaskManager.GcJobSummary - 169 total (1 active), Execution time: mean = 12.101 us, total = 2.045 ms, Queueing time: mean = 79.844 us, max = 171.691 us, min = 27.203 us, total = 13.494 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:15:53,826 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4249 total (25 active) Queueing time: mean = 20.823 us, max = 333.684 us, min = 2.330 us, total = 88.477 ms Execution time: mean = 3.427 s, total = 14560.135 s Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 30.514 us, total = 102.557 ms, Queueing time: mean = 11.285 us, max = 291.078 us, min = 2.330 us, total = 37.928 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 232 total (0 active), Execution time: mean = 44.084 us, total = 10.227 ms, Queueing time: mean = 111.393 us, max = 333.684 us, min = 14.442 us, total = 25.843 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 232 total (24 active), Execution time: mean = 62.758 s, total = 14559.858 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 185 total (0 active), Execution time: mean = 120.200 us, total = 22.237 ms, Queueing time: mean = 114.450 us, max = 247.252 us, min = 23.656 us, total = 21.173 ms InternalPubSubGcsService.grpc_server.GcsPublish - 185 total (0 active), Execution time: mean = 668.002 us, total = 123.580 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 3 total (1 active), Execution time: mean = 600.277 us, total = 1.801 ms, Queueing time: mean = 38.188 us, max = 62.049 us, min = 52.516 us, total = 114.565 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:15:53,826 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3387 total (0 active) Queueing time: mean = 11.196 us, max = 1.423 ms, min = 568.000 ns, total = 37.922 ms Execution time: mean = 28.261 us, total = 95.722 ms Event stats: event_loop_lag_probe - 3361 total (0 active), Execution time: mean = 28.019 us, total = 94.173 ms, Queueing time: mean = 10.724 us, max = 1.423 ms, min = 1.850 us, total = 36.042 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:16:53,827 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 180 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 197 - cumulative published bytes: 151379 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:16:53,828 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 11217 total (4 active) Queueing time: mean = 267.406 us, max = 279.303 ms, min = 807.000 ns, total = 2.999 s Execution time: mean = 263.576 us, total = 2.957 s Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 24.123 us, total = 86.867 ms, Queueing time: mean = 13.041 us, max = 13.097 ms, min = 2.088 us, total = 46.959 ms GcsInMemoryStore.Put - 1486 total (0 active), Execution time: mean = 194.153 us, total = 288.512 ms, Queueing time: mean = 1.359 ms, max = 278.545 ms, min = 2.310 us, total = 2.019 s RayletLoadPulled - 900 total (1 active), Execution time: mean = 237.492 us, total = 213.743 ms, Queueing time: mean = 74.118 us, max = 266.426 us, min = 11.570 us, total = 66.706 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 897 total (0 active), Execution time: mean = 75.586 us, total = 67.801 ms, Queueing time: mean = 104.877 us, max = 269.272 us, min = 12.278 us, total = 94.075 ms NodeManagerService.grpc_client.GetResourceLoad - 897 total (0 active), Execution time: mean = 1.473 ms, total = 1.321 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 374 total (0 active), Execution time: mean = 40.675 us, total = 15.212 ms, Queueing time: mean = 101.370 us, max = 200.615 us, min = 11.443 us, total = 37.912 ms InternalKVGcsService.grpc_server.InternalKVPut - 374 total (0 active), Execution time: mean = 567.705 us, total = 212.322 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 360 total (0 active), Execution time: mean = 32.783 us, total = 11.802 ms, Queueing time: mean = 118.605 us, max = 233.188 us, min = 20.329 us, total = 42.698 ms NodeInfoGcsService.grpc_server.CheckAlive - 360 total (0 active), Execution time: mean = 589.146 us, total = 212.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 301 total (1 active), Execution time: mean = 9.981 us, total = 3.004 ms, Queueing time: mean = 76.584 us, max = 232.777 us, min = 19.812 us, total = 23.052 ms HealthCheck - 298 total (0 active), Execution time: mean = 7.744 us, total = 2.308 ms, Queueing time: mean = 104.076 us, max = 399.856 us, min = 19.476 us, total = 31.015 ms GcsInMemoryStore.Get - 249 total (0 active), Execution time: mean = 24.930 us, total = 6.208 ms, Queueing time: mean = 14.115 us, max = 64.163 us, min = 4.068 us, total = 3.515 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 248 total (0 active), Execution time: mean = 51.999 us, total = 12.896 ms, Queueing time: mean = 110.184 us, max = 269.544 us, min = 13.883 us, total = 27.326 ms InternalKVGcsService.grpc_server.InternalKVGet - 248 total (0 active), Execution time: mean = 613.101 us, total = 152.049 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 180 total (0 active), Execution time: mean = 91.520 us, total = 16.474 ms, Queueing time: mean = 114.954 us, max = 296.469 us, min = 19.828 us, total = 20.692 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 180 total (0 active), Execution time: mean = 705.911 us, total = 127.064 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 90 total (1 active), Execution time: mean = 1.383 ms, total = 124.489 ms, Queueing time: mean = 71.900 us, max = 315.299 us, min = 17.370 us, total = 6.471 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 15 total (1 active, 1 running), Execution time: mean = 1.474 ms, total = 22.105 ms, Queueing time: mean = 53.571 us, max = 78.539 us, min = 19.831 us, total = 803.558 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:16:53,828 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 41457 total (1 active) Queueing time: mean = 50.659 us, max = 7.565 ms, min = 2.216 us, total = 2.100 s Execution time: mean = 259.440 us, total = 10.756 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 18837 total (0 active), Execution time: mean = 537.570 us, total = 10.126 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 18837 total (0 active), Execution time: mean = 27.944 us, total = 526.379 ms, Queueing time: mean = 108.862 us, max = 7.565 ms, min = 5.787 us, total = 2.051 s event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 27.999 us, total = 100.823 ms, Queueing time: mean = 9.721 us, max = 618.612 us, min = 2.216 us, total = 35.005 ms GcsTaskManager.GcJobSummary - 181 total (1 active), Execution time: mean = 12.127 us, total = 2.195 ms, Queueing time: mean = 79.774 us, max = 171.691 us, min = 27.203 us, total = 14.439 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:16:53,829 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4540 total (25 active) Queueing time: mean = 20.783 us, max = 333.684 us, min = 2.330 us, total = 94.353 ms Execution time: mean = 3.286 s, total = 14920.321 s Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 30.632 us, total = 110.305 ms, Queueing time: mean = 11.334 us, max = 291.078 us, min = 2.330 us, total = 40.812 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 245 total (0 active), Execution time: mean = 44.121 us, total = 10.810 ms, Queueing time: mean = 111.214 us, max = 333.684 us, min = 14.442 us, total = 27.247 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 245 total (24 active), Execution time: mean = 60.898 s, total = 14920.026 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 197 total (0 active), Execution time: mean = 119.961 us, total = 23.632 ms, Queueing time: mean = 114.764 us, max = 247.252 us, min = 23.656 us, total = 22.608 ms InternalPubSubGcsService.grpc_server.GcsPublish - 197 total (0 active), Execution time: mean = 668.030 us, total = 131.602 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 465.212 us, total = 1.861 ms, Queueing time: mean = 67.030 us, max = 153.556 us, min = 52.516 us, total = 268.121 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:16:53,829 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3627 total (0 active) Queueing time: mean = 11.094 us, max = 1.423 ms, min = 568.000 ns, total = 40.239 ms Execution time: mean = 28.162 us, total = 102.145 ms Event stats: event_loop_lag_probe - 3601 total (0 active), Execution time: mean = 27.935 us, total = 100.596 ms, Queueing time: mean = 10.653 us, max = 1.423 ms, min = 1.850 us, total = 38.360 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:17:53,829 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 192 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 209 - cumulative published bytes: 161207 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:17:53,830 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 11936 total (4 active) Queueing time: mean = 253.975 us, max = 279.303 ms, min = 807.000 ns, total = 3.031 s Execution time: mean = 262.570 us, total = 3.134 s Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 24.188 us, total = 92.906 ms, Queueing time: mean = 12.832 us, max = 13.097 ms, min = 2.088 us, total = 49.289 ms GcsInMemoryStore.Put - 1582 total (0 active), Execution time: mean = 182.691 us, total = 289.017 ms, Queueing time: mean = 1.279 ms, max = 278.545 ms, min = 2.310 us, total = 2.023 s RayletLoadPulled - 960 total (1 active), Execution time: mean = 237.683 us, total = 228.176 ms, Queueing time: mean = 74.775 us, max = 266.426 us, min = 11.570 us, total = 71.784 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 957 total (0 active), Execution time: mean = 75.953 us, total = 72.687 ms, Queueing time: mean = 105.335 us, max = 269.272 us, min = 12.278 us, total = 100.806 ms NodeManagerService.grpc_client.GetResourceLoad - 957 total (0 active), Execution time: mean = 1.476 ms, total = 1.412 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 398 total (0 active), Execution time: mean = 40.726 us, total = 16.209 ms, Queueing time: mean = 103.024 us, max = 200.615 us, min = 11.443 us, total = 41.004 ms InternalKVGcsService.grpc_server.InternalKVPut - 398 total (0 active), Execution time: mean = 569.300 us, total = 226.582 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 384 total (0 active), Execution time: mean = 33.233 us, total = 12.761 ms, Queueing time: mean = 118.521 us, max = 233.188 us, min = 20.329 us, total = 45.512 ms NodeInfoGcsService.grpc_server.CheckAlive - 384 total (0 active), Execution time: mean = 591.953 us, total = 227.310 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 321 total (1 active), Execution time: mean = 9.997 us, total = 3.209 ms, Queueing time: mean = 76.770 us, max = 232.777 us, min = 19.812 us, total = 24.643 ms HealthCheck - 318 total (0 active), Execution time: mean = 7.753 us, total = 2.465 ms, Queueing time: mean = 104.982 us, max = 399.856 us, min = 19.476 us, total = 33.384 ms GcsInMemoryStore.Get - 261 total (0 active), Execution time: mean = 24.718 us, total = 6.451 ms, Queueing time: mean = 14.050 us, max = 64.163 us, min = 4.068 us, total = 3.667 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 260 total (0 active), Execution time: mean = 52.180 us, total = 13.567 ms, Queueing time: mean = 110.661 us, max = 269.544 us, min = 13.883 us, total = 28.772 ms InternalKVGcsService.grpc_server.InternalKVGet - 260 total (0 active), Execution time: mean = 613.505 us, total = 159.511 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 192 total (0 active), Execution time: mean = 91.975 us, total = 17.659 ms, Queueing time: mean = 115.622 us, max = 296.469 us, min = 19.828 us, total = 22.199 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 192 total (0 active), Execution time: mean = 704.030 us, total = 135.174 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 96 total (1 active), Execution time: mean = 1.390 ms, total = 133.409 ms, Queueing time: mean = 73.068 us, max = 315.299 us, min = 17.370 us, total = 7.015 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 16 total (1 active, 1 running), Execution time: mean = 1.531 ms, total = 24.499 ms, Queueing time: mean = 51.406 us, max = 78.539 us, min = 18.942 us, total = 822.500 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:17:53,830 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 44229 total (1 active) Queueing time: mean = 50.757 us, max = 7.565 ms, min = 2.216 us, total = 2.245 s Execution time: mean = 259.692 us, total = 11.486 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 20097 total (0 active), Execution time: mean = 538.167 us, total = 10.816 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 20097 total (0 active), Execution time: mean = 27.861 us, total = 559.921 ms, Queueing time: mean = 109.061 us, max = 7.565 ms, min = 5.787 us, total = 2.192 s event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 28.151 us, total = 108.130 ms, Queueing time: mean = 9.781 us, max = 618.612 us, min = 2.216 us, total = 37.568 ms GcsTaskManager.GcJobSummary - 193 total (1 active), Execution time: mean = 12.067 us, total = 2.329 ms, Queueing time: mean = 80.069 us, max = 171.691 us, min = 27.203 us, total = 15.453 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:17:53,830 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 4828 total (25 active) Queueing time: mean = 20.759 us, max = 333.684 us, min = 2.330 us, total = 100.224 ms Execution time: mean = 3.103 s, total = 14980.490 s Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 30.847 us, total = 118.484 ms, Queueing time: mean = 11.395 us, max = 291.078 us, min = 2.330 us, total = 43.769 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 257 total (0 active), Execution time: mean = 44.248 us, total = 11.372 ms, Queueing time: mean = 111.251 us, max = 333.684 us, min = 14.442 us, total = 28.591 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 257 total (24 active), Execution time: mean = 58.289 s, total = 14980.176 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 209 total (0 active), Execution time: mean = 120.522 us, total = 25.189 ms, Queueing time: mean = 115.684 us, max = 247.252 us, min = 23.656 us, total = 24.178 ms InternalPubSubGcsService.grpc_server.GcsPublish - 209 total (0 active), Execution time: mean = 670.298 us, total = 140.092 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 465.212 us, total = 1.861 ms, Queueing time: mean = 67.030 us, max = 153.556 us, min = 52.516 us, total = 268.121 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:17:53,831 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 3867 total (0 active) Queueing time: mean = 11.040 us, max = 1.423 ms, min = 568.000 ns, total = 42.693 ms Execution time: mean = 28.160 us, total = 108.894 ms Event stats: event_loop_lag_probe - 3841 total (0 active), Execution time: mean = 27.947 us, total = 107.345 ms, Queueing time: mean = 10.626 us, max = 1.423 ms, min = 1.850 us, total = 40.814 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:18:53,831 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 204 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 221 - cumulative published bytes: 171035 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:18:53,832 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 12655 total (4 active) Queueing time: mean = 241.608 us, max = 279.303 ms, min = 807.000 ns, total = 3.058 s Execution time: mean = 260.729 us, total = 3.300 s Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 24.167 us, total = 98.625 ms, Queueing time: mean = 12.593 us, max = 13.097 ms, min = 2.088 us, total = 51.391 ms GcsInMemoryStore.Put - 1678 total (0 active), Execution time: mean = 172.495 us, total = 289.446 ms, Queueing time: mean = 1.208 ms, max = 278.545 ms, min = 2.310 us, total = 2.027 s RayletLoadPulled - 1020 total (1 active), Execution time: mean = 236.796 us, total = 241.532 ms, Queueing time: mean = 74.220 us, max = 266.426 us, min = 11.570 us, total = 75.705 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1017 total (0 active), Execution time: mean = 75.984 us, total = 77.276 ms, Queueing time: mean = 104.712 us, max = 269.272 us, min = 12.278 us, total = 106.492 ms NodeManagerService.grpc_client.GetResourceLoad - 1017 total (0 active), Execution time: mean = 1.474 ms, total = 1.499 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 422 total (0 active), Execution time: mean = 40.796 us, total = 17.216 ms, Queueing time: mean = 102.092 us, max = 200.615 us, min = 11.443 us, total = 43.083 ms InternalKVGcsService.grpc_server.InternalKVPut - 422 total (0 active), Execution time: mean = 568.451 us, total = 239.886 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 408 total (0 active), Execution time: mean = 33.735 us, total = 13.764 ms, Queueing time: mean = 117.377 us, max = 233.188 us, min = 20.329 us, total = 47.890 ms NodeInfoGcsService.grpc_server.CheckAlive - 408 total (0 active), Execution time: mean = 589.725 us, total = 240.608 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 341 total (1 active), Execution time: mean = 9.978 us, total = 3.402 ms, Queueing time: mean = 76.803 us, max = 232.777 us, min = 19.812 us, total = 26.190 ms HealthCheck - 338 total (0 active), Execution time: mean = 7.699 us, total = 2.602 ms, Queueing time: mean = 103.788 us, max = 399.856 us, min = 19.476 us, total = 35.080 ms GcsInMemoryStore.Get - 273 total (0 active), Execution time: mean = 24.384 us, total = 6.657 ms, Queueing time: mean = 13.903 us, max = 64.163 us, min = 4.068 us, total = 3.795 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 272 total (0 active), Execution time: mean = 51.883 us, total = 14.112 ms, Queueing time: mean = 109.197 us, max = 269.544 us, min = 13.883 us, total = 29.702 ms InternalKVGcsService.grpc_server.InternalKVGet - 272 total (0 active), Execution time: mean = 610.292 us, total = 165.999 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 204 total (0 active), Execution time: mean = 90.889 us, total = 18.541 ms, Queueing time: mean = 113.828 us, max = 296.469 us, min = 19.828 us, total = 23.221 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 204 total (0 active), Execution time: mean = 698.591 us, total = 142.513 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 102 total (1 active), Execution time: mean = 1.390 ms, total = 141.805 ms, Queueing time: mean = 72.983 us, max = 315.299 us, min = 17.370 us, total = 7.444 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 17 total (1 active, 1 running), Execution time: mean = 1.546 ms, total = 26.282 ms, Queueing time: mean = 51.574 us, max = 78.539 us, min = 18.942 us, total = 876.765 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:18:53,832 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 47001 total (1 active) Queueing time: mean = 50.662 us, max = 7.565 ms, min = 2.216 us, total = 2.381 s Execution time: mean = 259.483 us, total = 12.196 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 21357 total (0 active), Execution time: mean = 537.773 us, total = 11.485 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 21357 total (0 active), Execution time: mean = 27.732 us, total = 592.269 ms, Queueing time: mean = 108.852 us, max = 7.565 ms, min = 5.193 us, total = 2.325 s event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 28.426 us, total = 116.006 ms, Queueing time: mean = 9.787 us, max = 618.612 us, min = 2.216 us, total = 39.942 ms GcsTaskManager.GcJobSummary - 205 total (1 active), Execution time: mean = 11.984 us, total = 2.457 ms, Queueing time: mean = 79.831 us, max = 171.691 us, min = 27.203 us, total = 16.365 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:18:53,832 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5116 total (25 active) Queueing time: mean = 20.690 us, max = 333.684 us, min = 2.330 us, total = 105.848 ms Execution time: mean = 2.940 s, total = 15040.670 s Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 30.827 us, total = 125.806 ms, Queueing time: mean = 11.417 us, max = 291.078 us, min = 2.330 us, total = 46.592 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 269 total (0 active), Execution time: mean = 44.713 us, total = 12.028 ms, Queueing time: mean = 111.353 us, max = 333.684 us, min = 14.442 us, total = 29.954 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 269 total (24 active), Execution time: mean = 55.912 s, total = 15040.339 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 221 total (0 active), Execution time: mean = 120.141 us, total = 26.551 ms, Queueing time: mean = 115.912 us, max = 247.252 us, min = 23.656 us, total = 25.617 ms InternalPubSubGcsService.grpc_server.GcsPublish - 221 total (0 active), Execution time: mean = 669.770 us, total = 148.019 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 465.212 us, total = 1.861 ms, Queueing time: mean = 67.030 us, max = 153.556 us, min = 52.516 us, total = 268.121 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:18:53,832 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4107 total (0 active) Queueing time: mean = 10.970 us, max = 1.423 ms, min = 568.000 ns, total = 45.053 ms Execution time: mean = 28.048 us, total = 115.195 ms Event stats: event_loop_lag_probe - 4081 total (0 active), Execution time: mean = 27.848 us, total = 113.646 ms, Queueing time: mean = 10.579 us, max = 1.423 ms, min = 1.850 us, total = 43.173 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:19:53,832 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 216 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 234 - cumulative published bytes: 180897 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:19:53,833 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 13374 total (4 active) Queueing time: mean = 230.688 us, max = 279.303 ms, min = 807.000 ns, total = 3.085 s Execution time: mean = 259.521 us, total = 3.471 s Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 24.178 us, total = 104.473 ms, Queueing time: mean = 12.409 us, max = 13.097 ms, min = 2.088 us, total = 53.618 ms GcsInMemoryStore.Put - 1774 total (0 active), Execution time: mean = 163.428 us, total = 289.921 ms, Queueing time: mean = 1.145 ms, max = 278.545 ms, min = 2.310 us, total = 2.032 s RayletLoadPulled - 1080 total (1 active), Execution time: mean = 237.522 us, total = 256.524 ms, Queueing time: mean = 74.143 us, max = 266.426 us, min = 11.570 us, total = 80.075 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1077 total (0 active), Execution time: mean = 76.046 us, total = 81.902 ms, Queueing time: mean = 104.045 us, max = 269.272 us, min = 12.278 us, total = 112.056 ms NodeManagerService.grpc_client.GetResourceLoad - 1077 total (0 active), Execution time: mean = 1.474 ms, total = 1.587 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 446 total (0 active), Execution time: mean = 40.926 us, total = 18.253 ms, Queueing time: mean = 101.394 us, max = 228.583 us, min = 11.443 us, total = 45.222 ms InternalKVGcsService.grpc_server.InternalKVPut - 446 total (0 active), Execution time: mean = 568.223 us, total = 253.428 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 432 total (0 active), Execution time: mean = 33.702 us, total = 14.559 ms, Queueing time: mean = 116.505 us, max = 233.188 us, min = 20.329 us, total = 50.330 ms NodeInfoGcsService.grpc_server.CheckAlive - 432 total (0 active), Execution time: mean = 588.869 us, total = 254.391 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 361 total (1 active), Execution time: mean = 9.941 us, total = 3.589 ms, Queueing time: mean = 76.321 us, max = 232.777 us, min = 19.812 us, total = 27.552 ms HealthCheck - 358 total (0 active), Execution time: mean = 7.679 us, total = 2.749 ms, Queueing time: mean = 103.794 us, max = 399.856 us, min = 19.476 us, total = 37.158 ms GcsInMemoryStore.Get - 285 total (0 active), Execution time: mean = 24.138 us, total = 6.879 ms, Queueing time: mean = 13.840 us, max = 64.163 us, min = 4.068 us, total = 3.944 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 284 total (0 active), Execution time: mean = 52.012 us, total = 14.771 ms, Queueing time: mean = 108.562 us, max = 269.544 us, min = 13.883 us, total = 30.832 ms InternalKVGcsService.grpc_server.InternalKVGet - 284 total (0 active), Execution time: mean = 609.307 us, total = 173.043 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 216 total (0 active), Execution time: mean = 90.958 us, total = 19.647 ms, Queueing time: mean = 114.028 us, max = 296.469 us, min = 19.828 us, total = 24.630 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 216 total (0 active), Execution time: mean = 699.647 us, total = 151.124 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 108 total (1 active), Execution time: mean = 1.391 ms, total = 150.227 ms, Queueing time: mean = 72.641 us, max = 315.299 us, min = 17.370 us, total = 7.845 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 18 total (1 active, 1 running), Execution time: mean = 1.543 ms, total = 27.776 ms, Queueing time: mean = 51.881 us, max = 78.539 us, min = 18.942 us, total = 933.853 us GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:19:53,833 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 49773 total (1 active) Queueing time: mean = 50.676 us, max = 7.565 ms, min = 2.216 us, total = 2.522 s Execution time: mean = 259.384 us, total = 12.910 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 22617 total (0 active), Execution time: mean = 537.566 us, total = 12.158 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 22617 total (0 active), Execution time: mean = 27.691 us, total = 626.289 ms, Queueing time: mean = 108.875 us, max = 7.565 ms, min = 5.193 us, total = 2.462 s event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 28.540 us, total = 123.321 ms, Queueing time: mean = 9.846 us, max = 618.612 us, min = 2.216 us, total = 42.543 ms GcsTaskManager.GcJobSummary - 217 total (1 active), Execution time: mean = 11.864 us, total = 2.574 ms, Queueing time: mean = 79.452 us, max = 171.691 us, min = 27.203 us, total = 17.241 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:19:53,834 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5408 total (25 active) Queueing time: mean = 20.543 us, max = 333.684 us, min = 2.330 us, total = 111.097 ms Execution time: mean = 2.792 s, total = 15100.809 s Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 30.773 us, total = 132.971 ms, Queueing time: mean = 11.376 us, max = 291.078 us, min = 2.330 us, total = 49.154 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 282 total (0 active), Execution time: mean = 44.605 us, total = 12.579 ms, Queueing time: mean = 110.650 us, max = 333.684 us, min = 14.442 us, total = 31.203 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 282 total (24 active), Execution time: mean = 53.548 s, total = 15100.460 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 234 total (0 active), Execution time: mean = 119.271 us, total = 27.909 ms, Queueing time: mean = 115.617 us, max = 247.252 us, min = 23.656 us, total = 27.054 ms InternalPubSubGcsService.grpc_server.GcsPublish - 234 total (0 active), Execution time: mean = 669.196 us, total = 156.592 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 465.212 us, total = 1.861 ms, Queueing time: mean = 67.030 us, max = 153.556 us, min = 52.516 us, total = 268.121 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:19:53,834 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4347 total (0 active) Queueing time: mean = 10.943 us, max = 1.423 ms, min = 568.000 ns, total = 47.570 ms Execution time: mean = 28.031 us, total = 121.849 ms Event stats: event_loop_lag_probe - 4321 total (0 active), Execution time: mean = 27.841 us, total = 120.300 ms, Queueing time: mean = 10.574 us, max = 1.423 ms, min = 1.850 us, total = 45.691 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:20:53,834 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 228 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 247 - cumulative published bytes: 190759 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:20:53,835 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 14093 total (4 active) Queueing time: mean = 221.214 us, max = 279.303 ms, min = 807.000 ns, total = 3.118 s Execution time: mean = 258.913 us, total = 3.649 s Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 24.136 us, total = 110.087 ms, Queueing time: mean = 12.226 us, max = 13.097 ms, min = 2.088 us, total = 55.764 ms GcsInMemoryStore.Put - 1870 total (0 active), Execution time: mean = 155.277 us, total = 290.369 ms, Queueing time: mean = 1.089 ms, max = 278.545 ms, min = 2.310 us, total = 2.036 s RayletLoadPulled - 1140 total (1 active), Execution time: mean = 239.059 us, total = 272.527 ms, Queueing time: mean = 74.766 us, max = 266.426 us, min = 11.570 us, total = 85.234 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1137 total (0 active), Execution time: mean = 76.226 us, total = 86.669 ms, Queueing time: mean = 105.515 us, max = 269.272 us, min = 12.278 us, total = 119.970 ms NodeManagerService.grpc_client.GetResourceLoad - 1137 total (0 active), Execution time: mean = 1.479 ms, total = 1.681 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 470 total (0 active), Execution time: mean = 40.492 us, total = 19.031 ms, Queueing time: mean = 101.288 us, max = 228.583 us, min = 11.443 us, total = 47.605 ms InternalKVGcsService.grpc_server.InternalKVPut - 470 total (0 active), Execution time: mean = 567.076 us, total = 266.526 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 456 total (0 active), Execution time: mean = 33.741 us, total = 15.386 ms, Queueing time: mean = 117.037 us, max = 233.188 us, min = 20.329 us, total = 53.369 ms NodeInfoGcsService.grpc_server.CheckAlive - 456 total (0 active), Execution time: mean = 590.242 us, total = 269.150 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 381 total (1 active), Execution time: mean = 9.847 us, total = 3.752 ms, Queueing time: mean = 76.078 us, max = 232.777 us, min = 19.812 us, total = 28.986 ms HealthCheck - 378 total (0 active), Execution time: mean = 7.708 us, total = 2.914 ms, Queueing time: mean = 104.563 us, max = 399.856 us, min = 19.476 us, total = 39.525 ms GcsInMemoryStore.Get - 297 total (0 active), Execution time: mean = 23.899 us, total = 7.098 ms, Queueing time: mean = 13.783 us, max = 64.163 us, min = 4.068 us, total = 4.094 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 296 total (0 active), Execution time: mean = 51.978 us, total = 15.386 ms, Queueing time: mean = 108.311 us, max = 269.544 us, min = 13.883 us, total = 32.060 ms InternalKVGcsService.grpc_server.InternalKVGet - 296 total (0 active), Execution time: mean = 609.278 us, total = 180.346 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 228 total (0 active), Execution time: mean = 90.575 us, total = 20.651 ms, Queueing time: mean = 114.597 us, max = 296.469 us, min = 19.828 us, total = 26.128 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 228 total (0 active), Execution time: mean = 700.207 us, total = 159.647 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 114 total (1 active), Execution time: mean = 1.392 ms, total = 158.667 ms, Queueing time: mean = 72.295 us, max = 315.299 us, min = 17.370 us, total = 8.242 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 19 total (1 active, 1 running), Execution time: mean = 1.542 ms, total = 29.296 ms, Queueing time: mean = 54.869 us, max = 108.649 us, min = 18.942 us, total = 1.043 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:20:53,835 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 52545 total (1 active) Queueing time: mean = 50.794 us, max = 7.565 ms, min = 2.216 us, total = 2.669 s Execution time: mean = 259.459 us, total = 13.633 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 23877 total (0 active), Execution time: mean = 537.779 us, total = 12.841 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 23877 total (0 active), Execution time: mean = 27.637 us, total = 659.888 ms, Queueing time: mean = 109.123 us, max = 7.565 ms, min = 5.193 us, total = 2.606 s event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 28.525 us, total = 130.104 ms, Queueing time: mean = 9.858 us, max = 618.612 us, min = 2.216 us, total = 44.964 ms GcsTaskManager.GcJobSummary - 229 total (1 active), Execution time: mean = 11.848 us, total = 2.713 ms, Queueing time: mean = 80.219 us, max = 229.756 us, min = 27.203 us, total = 18.370 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:20:53,835 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 5700 total (25 active) Queueing time: mean = 20.488 us, max = 333.684 us, min = 2.330 us, total = 116.783 ms Execution time: mean = 2.660 s, total = 15161.064 s Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 30.831 us, total = 140.620 ms, Queueing time: mean = 11.397 us, max = 291.078 us, min = 2.330 us, total = 51.982 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 295 total (0 active), Execution time: mean = 44.507 us, total = 13.130 ms, Queueing time: mean = 110.629 us, max = 333.684 us, min = 14.442 us, total = 32.636 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 295 total (24 active), Execution time: mean = 51.392 s, total = 15160.698 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 247 total (0 active), Execution time: mean = 118.458 us, total = 29.259 ms, Queueing time: mean = 115.304 us, max = 247.252 us, min = 23.656 us, total = 28.480 ms InternalPubSubGcsService.grpc_server.GcsPublish - 247 total (0 active), Execution time: mean = 668.281 us, total = 165.065 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 4 total (1 active), Execution time: mean = 465.212 us, total = 1.861 ms, Queueing time: mean = 67.030 us, max = 153.556 us, min = 52.516 us, total = 268.121 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:20:53,835 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4587 total (0 active) Queueing time: mean = 10.907 us, max = 1.423 ms, min = 568.000 ns, total = 50.029 ms Execution time: mean = 27.996 us, total = 128.419 ms Event stats: event_loop_lag_probe - 4561 total (0 active), Execution time: mean = 27.816 us, total = 126.869 ms, Queueing time: mean = 10.557 us, max = 1.423 ms, min = 1.850 us, total = 48.150 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:21:53,836 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 240 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 259 - cumulative published bytes: 200587 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:21:53,836 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 14811 total (4 active) Queueing time: mean = 212.689 us, max = 279.303 ms, min = 807.000 ns, total = 3.150 s Execution time: mean = 258.872 us, total = 3.834 s Event stats: event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 24.150 us, total = 115.919 ms, Queueing time: mean = 12.072 us, max = 13.097 ms, min = 2.088 us, total = 57.946 ms GcsInMemoryStore.Put - 1966 total (0 active), Execution time: mean = 147.957 us, total = 290.884 ms, Queueing time: mean = 1.038 ms, max = 278.545 ms, min = 2.310 us, total = 2.040 s RayletLoadPulled - 1200 total (1 active), Execution time: mean = 239.909 us, total = 287.891 ms, Queueing time: mean = 75.165 us, max = 266.426 us, min = 11.570 us, total = 90.198 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1197 total (0 active), Execution time: mean = 76.325 us, total = 91.361 ms, Queueing time: mean = 106.762 us, max = 269.272 us, min = 12.278 us, total = 127.794 ms NodeManagerService.grpc_client.GetResourceLoad - 1197 total (0 active), Execution time: mean = 1.486 ms, total = 1.778 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 494 total (0 active), Execution time: mean = 40.460 us, total = 19.987 ms, Queueing time: mean = 102.032 us, max = 228.583 us, min = 11.443 us, total = 50.404 ms InternalKVGcsService.grpc_server.InternalKVPut - 494 total (0 active), Execution time: mean = 568.822 us, total = 280.998 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 480 total (0 active), Execution time: mean = 33.760 us, total = 16.205 ms, Queueing time: mean = 117.676 us, max = 233.188 us, min = 20.329 us, total = 56.484 ms NodeInfoGcsService.grpc_server.CheckAlive - 480 total (0 active), Execution time: mean = 592.609 us, total = 284.452 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 401 total (1 active), Execution time: mean = 9.811 us, total = 3.934 ms, Queueing time: mean = 76.036 us, max = 232.777 us, min = 19.812 us, total = 30.491 ms HealthCheck - 398 total (0 active), Execution time: mean = 7.708 us, total = 3.068 ms, Queueing time: mean = 104.763 us, max = 399.856 us, min = 19.476 us, total = 41.696 ms GcsInMemoryStore.Get - 309 total (0 active), Execution time: mean = 23.764 us, total = 7.343 ms, Queueing time: mean = 13.763 us, max = 64.163 us, min = 4.068 us, total = 4.253 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 308 total (0 active), Execution time: mean = 52.128 us, total = 16.056 ms, Queueing time: mean = 108.654 us, max = 269.544 us, min = 13.883 us, total = 33.465 ms InternalKVGcsService.grpc_server.InternalKVGet - 308 total (0 active), Execution time: mean = 610.407 us, total = 188.005 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 240 total (0 active), Execution time: mean = 91.350 us, total = 21.924 ms, Queueing time: mean = 115.533 us, max = 296.469 us, min = 19.828 us, total = 27.728 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 240 total (0 active), Execution time: mean = 703.691 us, total = 168.886 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 120 total (1 active), Execution time: mean = 1.399 ms, total = 167.872 ms, Queueing time: mean = 72.961 us, max = 315.299 us, min = 17.370 us, total = 8.755 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GCSServer.deadline_timer.debug_state_event_stats_print - 20 total (1 active, 1 running), Execution time: mean = 1.541 ms, total = 30.813 ms, Queueing time: mean = 55.201 us, max = 108.649 us, min = 18.942 us, total = 1.104 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:21:53,837 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 55316 total (1 active) Queueing time: mean = 50.989 us, max = 7.565 ms, min = 2.216 us, total = 2.821 s Execution time: mean = 259.786 us, total = 14.370 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 25137 total (0 active), Execution time: mean = 538.559 us, total = 13.538 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 25137 total (0 active), Execution time: mean = 27.557 us, total = 692.711 ms, Queueing time: mean = 109.348 us, max = 7.565 ms, min = 5.193 us, total = 2.749 s event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 28.539 us, total = 136.989 ms, Queueing time: mean = 10.908 us, max = 4.983 ms, min = 2.216 us, total = 52.361 ms GcsTaskManager.GcJobSummary - 241 total (1 active), Execution time: mean = 11.810 us, total = 2.846 ms, Queueing time: mean = 80.425 us, max = 229.756 us, min = 27.203 us, total = 19.383 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:21:53,837 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6032 total (25 active) Queueing time: mean = 20.689 us, max = 333.684 us, min = 2.330 us, total = 124.798 ms Execution time: mean = 4.712 s, total = 28421.230 s Event stats: event_loop_lag_probe - 4800 total (0 active), Execution time: mean = 30.945 us, total = 148.536 ms, Queueing time: mean = 11.443 us, max = 291.078 us, min = 2.330 us, total = 54.927 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 329 total (0 active), Execution time: mean = 42.985 us, total = 14.142 ms, Queueing time: mean = 110.870 us, max = 333.684 us, min = 14.442 us, total = 36.476 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 329 total (24 active), Execution time: mean = 86.386 s, total = 28420.845 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 259 total (0 active), Execution time: mean = 117.699 us, total = 30.484 ms, Queueing time: mean = 114.289 us, max = 247.252 us, min = 23.656 us, total = 29.601 ms InternalPubSubGcsService.grpc_server.GcsPublish - 259 total (0 active), Execution time: mean = 666.294 us, total = 172.570 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 635.705 us, total = 3.179 ms, Queueing time: mean = 75.429 us, max = 153.556 us, min = 52.516 us, total = 377.147 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:21:53,837 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 4827 total (0 active) Queueing time: mean = 10.861 us, max = 1.423 ms, min = 568.000 ns, total = 52.424 ms Execution time: mean = 27.957 us, total = 134.951 ms Event stats: event_loop_lag_probe - 4801 total (0 active), Execution time: mean = 27.786 us, total = 133.402 ms, Queueing time: mean = 10.528 us, max = 1.423 ms, min = 1.850 us, total = 50.545 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:22:53,837 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 252 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 271 - cumulative published bytes: 210415 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:22:53,838 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 15530 total (4 active) Queueing time: mean = 205.004 us, max = 279.303 ms, min = 807.000 ns, total = 3.184 s Execution time: mean = 259.133 us, total = 4.024 s Event stats: event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 24.212 us, total = 122.026 ms, Queueing time: mean = 11.963 us, max = 13.097 ms, min = 2.088 us, total = 60.292 ms GcsInMemoryStore.Put - 2062 total (0 active), Execution time: mean = 141.309 us, total = 291.379 ms, Queueing time: mean = 991.546 us, max = 278.545 ms, min = 2.310 us, total = 2.045 s RayletLoadPulled - 1260 total (1 active), Execution time: mean = 242.348 us, total = 305.359 ms, Queueing time: mean = 76.060 us, max = 266.426 us, min = 11.570 us, total = 95.836 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1257 total (0 active), Execution time: mean = 76.722 us, total = 96.439 ms, Queueing time: mean = 107.884 us, max = 269.272 us, min = 12.278 us, total = 135.610 ms NodeManagerService.grpc_client.GetResourceLoad - 1257 total (0 active), Execution time: mean = 1.495 ms, total = 1.880 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 518 total (0 active), Execution time: mean = 40.964 us, total = 21.219 ms, Queueing time: mean = 102.779 us, max = 228.583 us, min = 11.443 us, total = 53.240 ms InternalKVGcsService.grpc_server.InternalKVPut - 518 total (0 active), Execution time: mean = 570.870 us, total = 295.711 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 504 total (0 active), Execution time: mean = 33.798 us, total = 17.034 ms, Queueing time: mean = 118.609 us, max = 233.188 us, min = 20.329 us, total = 59.779 ms NodeInfoGcsService.grpc_server.CheckAlive - 504 total (0 active), Execution time: mean = 593.906 us, total = 299.328 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 421 total (1 active), Execution time: mean = 9.795 us, total = 4.124 ms, Queueing time: mean = 76.239 us, max = 232.777 us, min = 19.812 us, total = 32.097 ms HealthCheck - 418 total (0 active), Execution time: mean = 7.711 us, total = 3.223 ms, Queueing time: mean = 105.041 us, max = 399.856 us, min = 19.476 us, total = 43.907 ms GcsInMemoryStore.Get - 321 total (0 active), Execution time: mean = 23.722 us, total = 7.615 ms, Queueing time: mean = 13.766 us, max = 64.163 us, min = 4.068 us, total = 4.419 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 320 total (0 active), Execution time: mean = 52.452 us, total = 16.785 ms, Queueing time: mean = 108.942 us, max = 269.544 us, min = 13.883 us, total = 34.861 ms InternalKVGcsService.grpc_server.InternalKVGet - 320 total (0 active), Execution time: mean = 610.790 us, total = 195.453 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 252 total (0 active), Execution time: mean = 91.865 us, total = 23.150 ms, Queueing time: mean = 116.115 us, max = 296.469 us, min = 19.828 us, total = 29.261 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 252 total (0 active), Execution time: mean = 702.464 us, total = 177.021 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 126 total (1 active), Execution time: mean = 1.401 ms, total = 176.531 ms, Queueing time: mean = 73.605 us, max = 315.299 us, min = 17.370 us, total = 9.274 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 21 total (1 active, 1 running), Execution time: mean = 1.538 ms, total = 32.294 ms, Queueing time: mean = 56.299 us, max = 108.649 us, min = 18.942 us, total = 1.182 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:22:53,838 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 58088 total (1 active) Queueing time: mean = 51.012 us, max = 7.565 ms, min = 2.216 us, total = 2.963 s Execution time: mean = 259.954 us, total = 15.100 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 26397 total (0 active), Execution time: mean = 538.825 us, total = 14.223 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 26397 total (0 active), Execution time: mean = 27.634 us, total = 729.463 ms, Queueing time: mean = 109.394 us, max = 7.565 ms, min = 5.193 us, total = 2.888 s event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 28.646 us, total = 144.378 ms, Queueing time: mean = 10.927 us, max = 4.983 ms, min = 2.216 us, total = 55.071 ms GcsTaskManager.GcJobSummary - 253 total (1 active), Execution time: mean = 11.800 us, total = 2.985 ms, Queueing time: mean = 80.437 us, max = 229.756 us, min = 27.203 us, total = 20.350 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:22:53,838 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6320 total (25 active) Queueing time: mean = 20.582 us, max = 333.684 us, min = 2.330 us, total = 130.075 ms Execution time: mean = 4.507 s, total = 28481.434 s Event stats: event_loop_lag_probe - 5040 total (0 active), Execution time: mean = 31.082 us, total = 156.651 ms, Queueing time: mean = 11.464 us, max = 291.078 us, min = 2.330 us, total = 57.780 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 341 total (0 active), Execution time: mean = 43.106 us, total = 14.699 ms, Queueing time: mean = 110.358 us, max = 333.684 us, min = 14.442 us, total = 37.632 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 341 total (24 active), Execution time: mean = 83.522 s, total = 28481.030 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 271 total (0 active), Execution time: mean = 117.344 us, total = 31.800 ms, Queueing time: mean = 113.907 us, max = 247.252 us, min = 23.656 us, total = 30.869 ms InternalPubSubGcsService.grpc_server.GcsPublish - 271 total (0 active), Execution time: mean = 665.728 us, total = 180.412 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 635.705 us, total = 3.179 ms, Queueing time: mean = 75.429 us, max = 153.556 us, min = 52.516 us, total = 377.147 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:22:53,839 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5067 total (0 active) Queueing time: mean = 10.850 us, max = 1.423 ms, min = 568.000 ns, total = 54.979 ms Execution time: mean = 27.997 us, total = 141.860 ms Event stats: event_loop_lag_probe - 5041 total (0 active), Execution time: mean = 27.834 us, total = 140.311 ms, Queueing time: mean = 10.534 us, max = 1.423 ms, min = 1.850 us, total = 53.100 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:23:53,839 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 264 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 283 - cumulative published bytes: 220243 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:23:53,840 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 16249 total (4 active) Queueing time: mean = 197.901 us, max = 279.303 ms, min = -0.000 s, total = 3.216 s Execution time: mean = 258.567 us, total = 4.201 s Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 24.204 us, total = 127.800 ms, Queueing time: mean = 11.831 us, max = 13.097 ms, min = 2.088 us, total = 62.467 ms GcsInMemoryStore.Put - 2158 total (0 active), Execution time: mean = 135.299 us, total = 291.976 ms, Queueing time: mean = 949.460 us, max = 278.545 ms, min = 2.310 us, total = 2.049 s RayletLoadPulled - 1320 total (1 active), Execution time: mean = 242.645 us, total = 320.291 ms, Queueing time: mean = 76.436 us, max = 266.426 us, min = 11.570 us, total = 100.896 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1317 total (0 active), Execution time: mean = 77.029 us, total = 101.448 ms, Queueing time: mean = 108.730 us, max = 283.656 us, min = 12.278 us, total = 143.198 ms NodeManagerService.grpc_client.GetResourceLoad - 1317 total (0 active), Execution time: mean = 1.498 ms, total = 1.973 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 542 total (0 active), Execution time: mean = 40.956 us, total = 22.198 ms, Queueing time: mean = 102.914 us, max = 228.583 us, min = 11.443 us, total = 55.779 ms InternalKVGcsService.grpc_server.InternalKVPut - 542 total (0 active), Execution time: mean = 570.533 us, total = 309.229 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 528 total (0 active), Execution time: mean = 33.762 us, total = 17.826 ms, Queueing time: mean = 118.886 us, max = 233.188 us, min = 20.329 us, total = 62.772 ms NodeInfoGcsService.grpc_server.CheckAlive - 528 total (0 active), Execution time: mean = 594.188 us, total = 313.731 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 441 total (1 active), Execution time: mean = 9.782 us, total = 4.314 ms, Queueing time: mean = 76.196 us, max = 232.777 us, min = 19.812 us, total = 33.603 ms HealthCheck - 438 total (0 active), Execution time: mean = 7.719 us, total = 3.381 ms, Queueing time: mean = 105.621 us, max = 399.856 us, min = 19.476 us, total = 46.262 ms GcsInMemoryStore.Get - 333 total (0 active), Execution time: mean = 23.534 us, total = 7.837 ms, Queueing time: mean = 13.710 us, max = 64.163 us, min = 4.068 us, total = 4.565 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 332 total (0 active), Execution time: mean = 52.658 us, total = 17.482 ms, Queueing time: mean = 109.036 us, max = 269.544 us, min = 13.883 us, total = 36.200 ms InternalKVGcsService.grpc_server.InternalKVGet - 332 total (0 active), Execution time: mean = 610.361 us, total = 202.640 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 264 total (0 active), Execution time: mean = 91.920 us, total = 24.267 ms, Queueing time: mean = 116.734 us, max = 296.469 us, min = 19.828 us, total = 30.818 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 264 total (0 active), Execution time: mean = 701.657 us, total = 185.238 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 132 total (1 active), Execution time: mean = 1.399 ms, total = 184.699 ms, Queueing time: mean = 72.461 us, max = 315.299 us, min = -0.000 s, total = 9.565 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 22 total (1 active, 1 running), Execution time: mean = 1.540 ms, total = 33.890 ms, Queueing time: mean = 56.550 us, max = 108.649 us, min = 18.942 us, total = 1.244 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:23:53,840 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 60860 total (1 active) Queueing time: mean = 50.880 us, max = 7.565 ms, min = 2.216 us, total = 3.097 s Execution time: mean = 259.542 us, total = 15.796 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 27657 total (0 active), Execution time: mean = 537.954 us, total = 14.878 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 27657 total (0 active), Execution time: mean = 27.594 us, total = 763.180 ms, Queueing time: mean = 109.104 us, max = 7.565 ms, min = 5.193 us, total = 3.018 s event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 28.623 us, total = 151.130 ms, Queueing time: mean = 10.895 us, max = 4.983 ms, min = 2.216 us, total = 57.528 ms GcsTaskManager.GcJobSummary - 265 total (1 active), Execution time: mean = 12.005 us, total = 3.181 ms, Queueing time: mean = 80.784 us, max = 229.756 us, min = 27.203 us, total = 21.408 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:23:53,840 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6608 total (25 active) Queueing time: mean = 20.480 us, max = 333.684 us, min = 2.330 us, total = 135.329 ms Execution time: mean = 4.319 s, total = 28541.552 s Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 31.272 us, total = 165.117 ms, Queueing time: mean = 11.487 us, max = 291.078 us, min = 2.330 us, total = 60.653 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 353 total (0 active), Execution time: mean = 43.320 us, total = 15.292 ms, Queueing time: mean = 109.847 us, max = 333.684 us, min = 14.442 us, total = 38.776 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 353 total (24 active), Execution time: mean = 80.853 s, total = 28541.131 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 283 total (0 active), Execution time: mean = 116.976 us, total = 33.104 ms, Queueing time: mean = 113.447 us, max = 247.252 us, min = 23.656 us, total = 32.105 ms InternalPubSubGcsService.grpc_server.GcsPublish - 283 total (0 active), Execution time: mean = 664.137 us, total = 187.951 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 635.705 us, total = 3.179 ms, Queueing time: mean = 75.429 us, max = 153.556 us, min = 52.516 us, total = 377.147 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:23:53,840 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5306 total (0 active) Queueing time: mean = 10.833 us, max = 1.423 ms, min = 568.000 ns, total = 57.477 ms Execution time: mean = 28.009 us, total = 148.613 ms Event stats: event_loop_lag_probe - 5280 total (0 active), Execution time: mean = 27.853 us, total = 147.064 ms, Queueing time: mean = 10.530 us, max = 1.423 ms, min = 1.850 us, total = 55.598 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:24:53,840 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 276 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 295 - cumulative published bytes: 230071 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:24:53,841 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 16968 total (4 active) Queueing time: mean = 191.375 us, max = 279.303 ms, min = -0.000 s, total = 3.247 s Execution time: mean = 258.167 us, total = 4.381 s Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 24.149 us, total = 133.303 ms, Queueing time: mean = 11.697 us, max = 13.097 ms, min = 2.088 us, total = 64.569 ms GcsInMemoryStore.Put - 2254 total (0 active), Execution time: mean = 129.747 us, total = 292.449 ms, Queueing time: mean = 910.978 us, max = 278.545 ms, min = 2.310 us, total = 2.053 s RayletLoadPulled - 1380 total (1 active), Execution time: mean = 242.868 us, total = 335.158 ms, Queueing time: mean = 76.969 us, max = 266.426 us, min = 11.570 us, total = 106.217 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1377 total (0 active), Execution time: mean = 77.258 us, total = 106.384 ms, Queueing time: mean = 109.168 us, max = 283.656 us, min = 12.278 us, total = 150.325 ms NodeManagerService.grpc_client.GetResourceLoad - 1377 total (0 active), Execution time: mean = 1.502 ms, total = 2.068 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 566 total (0 active), Execution time: mean = 40.958 us, total = 23.182 ms, Queueing time: mean = 103.623 us, max = 228.583 us, min = 11.443 us, total = 58.651 ms InternalKVGcsService.grpc_server.InternalKVPut - 566 total (0 active), Execution time: mean = 571.951 us, total = 323.724 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 552 total (0 active), Execution time: mean = 33.817 us, total = 18.667 ms, Queueing time: mean = 118.598 us, max = 233.188 us, min = 20.329 us, total = 65.466 ms NodeInfoGcsService.grpc_server.CheckAlive - 552 total (0 active), Execution time: mean = 594.222 us, total = 328.011 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 461 total (1 active), Execution time: mean = 9.774 us, total = 4.506 ms, Queueing time: mean = 76.132 us, max = 232.777 us, min = 19.812 us, total = 35.097 ms HealthCheck - 458 total (0 active), Execution time: mean = 7.705 us, total = 3.529 ms, Queueing time: mean = 105.296 us, max = 399.856 us, min = 19.476 us, total = 48.226 ms GcsInMemoryStore.Get - 345 total (0 active), Execution time: mean = 23.362 us, total = 8.060 ms, Queueing time: mean = 13.693 us, max = 64.163 us, min = 4.068 us, total = 4.724 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 344 total (0 active), Execution time: mean = 52.812 us, total = 18.167 ms, Queueing time: mean = 109.348 us, max = 269.544 us, min = 13.883 us, total = 37.616 ms InternalKVGcsService.grpc_server.InternalKVGet - 344 total (0 active), Execution time: mean = 611.155 us, total = 210.237 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 276 total (0 active), Execution time: mean = 91.513 us, total = 25.258 ms, Queueing time: mean = 117.101 us, max = 296.469 us, min = 19.828 us, total = 32.320 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 276 total (0 active), Execution time: mean = 701.043 us, total = 193.488 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 138 total (1 active), Execution time: mean = 1.399 ms, total = 193.030 ms, Queueing time: mean = 72.517 us, max = 315.299 us, min = -0.000 s, total = 10.007 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 23 total (1 active, 1 running), Execution time: mean = 1.546 ms, total = 35.556 ms, Queueing time: mean = 56.847 us, max = 108.649 us, min = 18.942 us, total = 1.307 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:24:53,842 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 63632 total (1 active) Queueing time: mean = 50.987 us, max = 7.565 ms, min = 2.216 us, total = 3.244 s Execution time: mean = 259.762 us, total = 16.529 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 28917 total (0 active), Execution time: mean = 538.463 us, total = 15.571 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 28917 total (0 active), Execution time: mean = 27.566 us, total = 797.126 ms, Queueing time: mean = 109.342 us, max = 7.565 ms, min = 5.193 us, total = 3.162 s event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 28.621 us, total = 157.991 ms, Queueing time: mean = 10.867 us, max = 4.983 ms, min = 2.216 us, total = 59.989 ms GcsTaskManager.GcJobSummary - 277 total (1 active), Execution time: mean = 11.938 us, total = 3.307 ms, Queueing time: mean = 81.031 us, max = 229.756 us, min = 27.203 us, total = 22.445 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:24:53,842 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 6896 total (25 active) Queueing time: mean = 20.432 us, max = 333.684 us, min = 2.330 us, total = 140.897 ms Execution time: mean = 4.148 s, total = 28601.725 s Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 31.360 us, total = 173.106 ms, Queueing time: mean = 11.509 us, max = 291.078 us, min = 2.330 us, total = 63.531 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 365 total (0 active), Execution time: mean = 43.434 us, total = 15.853 ms, Queueing time: mean = 109.751 us, max = 333.684 us, min = 14.442 us, total = 40.059 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 365 total (24 active), Execution time: mean = 78.360 s, total = 28601.286 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 295 total (0 active), Execution time: mean = 116.817 us, total = 34.461 ms, Queueing time: mean = 113.603 us, max = 247.252 us, min = 23.656 us, total = 33.513 ms InternalPubSubGcsService.grpc_server.GcsPublish - 295 total (0 active), Execution time: mean = 663.257 us, total = 195.661 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 635.705 us, total = 3.179 ms, Queueing time: mean = 75.429 us, max = 153.556 us, min = 52.516 us, total = 377.147 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:24:53,842 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5546 total (0 active) Queueing time: mean = 10.794 us, max = 1.423 ms, min = 568.000 ns, total = 59.862 ms Execution time: mean = 27.957 us, total = 155.052 ms Event stats: event_loop_lag_probe - 5520 total (0 active), Execution time: mean = 27.809 us, total = 153.503 ms, Queueing time: mean = 10.504 us, max = 1.423 ms, min = 1.850 us, total = 57.982 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:25:53,842 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 288 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 309 - cumulative published bytes: 239967 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:25:53,843 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 17687 total (4 active) Queueing time: mean = 185.147 us, max = 279.303 ms, min = -0.000 s, total = 3.275 s Execution time: mean = 257.578 us, total = 4.556 s Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 24.134 us, total = 139.014 ms, Queueing time: mean = 11.576 us, max = 13.097 ms, min = 2.088 us, total = 66.678 ms GcsInMemoryStore.Put - 2350 total (0 active), Execution time: mean = 124.646 us, total = 292.917 ms, Queueing time: mean = 875.667 us, max = 278.545 ms, min = 2.310 us, total = 2.058 s RayletLoadPulled - 1440 total (1 active), Execution time: mean = 243.241 us, total = 350.267 ms, Queueing time: mean = 76.670 us, max = 266.426 us, min = 11.570 us, total = 110.405 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1437 total (0 active), Execution time: mean = 77.410 us, total = 111.238 ms, Queueing time: mean = 108.619 us, max = 283.656 us, min = 12.278 us, total = 156.086 ms NodeManagerService.grpc_client.GetResourceLoad - 1437 total (0 active), Execution time: mean = 1.503 ms, total = 2.160 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 590 total (0 active), Execution time: mean = 40.986 us, total = 24.182 ms, Queueing time: mean = 103.233 us, max = 228.583 us, min = 11.443 us, total = 60.907 ms InternalKVGcsService.grpc_server.InternalKVPut - 590 total (0 active), Execution time: mean = 572.236 us, total = 337.619 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 576 total (0 active), Execution time: mean = 33.762 us, total = 19.447 ms, Queueing time: mean = 117.864 us, max = 233.188 us, min = 20.329 us, total = 67.890 ms NodeInfoGcsService.grpc_server.CheckAlive - 576 total (0 active), Execution time: mean = 592.956 us, total = 341.543 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 481 total (1 active), Execution time: mean = 9.797 us, total = 4.713 ms, Queueing time: mean = 76.384 us, max = 232.777 us, min = 19.812 us, total = 36.741 ms HealthCheck - 478 total (0 active), Execution time: mean = 7.673 us, total = 3.668 ms, Queueing time: mean = 104.339 us, max = 399.856 us, min = 19.476 us, total = 49.874 ms GcsInMemoryStore.Get - 357 total (0 active), Execution time: mean = 23.195 us, total = 8.281 ms, Queueing time: mean = 13.662 us, max = 64.163 us, min = 4.068 us, total = 4.877 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 356 total (0 active), Execution time: mean = 53.057 us, total = 18.888 ms, Queueing time: mean = 108.632 us, max = 269.544 us, min = 13.883 us, total = 38.673 ms InternalKVGcsService.grpc_server.InternalKVGet - 356 total (0 active), Execution time: mean = 610.255 us, total = 217.251 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 288 total (0 active), Execution time: mean = 91.144 us, total = 26.249 ms, Queueing time: mean = 116.297 us, max = 296.469 us, min = 19.828 us, total = 33.493 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 288 total (0 active), Execution time: mean = 699.334 us, total = 201.408 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 144 total (1 active), Execution time: mean = 1.400 ms, total = 201.653 ms, Queueing time: mean = 72.889 us, max = 315.299 us, min = -0.000 s, total = 10.496 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 24 total (1 active, 1 running), Execution time: mean = 1.549 ms, total = 37.170 ms, Queueing time: mean = 57.192 us, max = 108.649 us, min = 18.942 us, total = 1.373 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:25:53,844 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 66404 total (1 active) Queueing time: mean = 50.907 us, max = 7.565 ms, min = 2.216 us, total = 3.380 s Execution time: mean = 259.533 us, total = 17.234 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 30177 total (0 active), Execution time: mean = 538.023 us, total = 16.236 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 30177 total (0 active), Execution time: mean = 27.517 us, total = 830.377 ms, Queueing time: mean = 109.178 us, max = 7.565 ms, min = 5.193 us, total = 3.295 s event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 28.521 us, total = 164.280 ms, Queueing time: mean = 10.813 us, max = 4.983 ms, min = 2.216 us, total = 62.281 ms GcsTaskManager.GcJobSummary - 289 total (1 active), Execution time: mean = 11.884 us, total = 3.435 ms, Queueing time: mean = 80.879 us, max = 229.756 us, min = 27.203 us, total = 23.374 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:25:53,844 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7192 total (25 active) Queueing time: mean = 20.336 us, max = 333.684 us, min = 2.330 us, total = 146.256 ms Execution time: mean = 3.985 s, total = 28661.936 s Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 31.269 us, total = 180.111 ms, Queueing time: mean = 11.483 us, max = 291.078 us, min = 2.330 us, total = 66.143 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 379 total (0 active), Execution time: mean = 43.594 us, total = 16.522 ms, Queueing time: mean = 109.510 us, max = 333.684 us, min = 14.442 us, total = 41.504 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 379 total (24 active), Execution time: mean = 75.624 s, total = 28661.479 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 309 total (0 active), Execution time: mean = 116.001 us, total = 35.844 ms, Queueing time: mean = 112.670 us, max = 247.252 us, min = 23.656 us, total = 34.815 ms InternalPubSubGcsService.grpc_server.GcsPublish - 309 total (0 active), Execution time: mean = 659.474 us, total = 203.777 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 5 total (1 active), Execution time: mean = 635.705 us, total = 3.179 ms, Queueing time: mean = 75.429 us, max = 153.556 us, min = 52.516 us, total = 377.147 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:25:53,844 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 5786 total (0 active) Queueing time: mean = 10.750 us, max = 1.423 ms, min = 568.000 ns, total = 62.197 ms Execution time: mean = 27.895 us, total = 161.398 ms Event stats: event_loop_lag_probe - 5760 total (0 active), Execution time: mean = 27.752 us, total = 159.849 ms, Queueing time: mean = 10.472 us, max = 1.423 ms, min = 1.850 us, total = 60.318 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:26:53,844 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 300 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 321 - cumulative published bytes: 249010 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:26:53,845 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 18406 total (4 active) Queueing time: mean = 179.348 us, max = 279.303 ms, min = -0.000 s, total = 3.301 s Execution time: mean = 257.091 us, total = 4.732 s Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 24.087 us, total = 144.519 ms, Queueing time: mean = 11.457 us, max = 13.097 ms, min = 2.088 us, total = 68.744 ms GcsInMemoryStore.Put - 2446 total (0 active), Execution time: mean = 119.946 us, total = 293.388 ms, Queueing time: mean = 843.134 us, max = 278.545 ms, min = 2.310 us, total = 2.062 s RayletLoadPulled - 1500 total (1 active), Execution time: mean = 243.302 us, total = 364.953 ms, Queueing time: mean = 76.392 us, max = 460.894 us, min = 11.570 us, total = 114.587 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1497 total (0 active), Execution time: mean = 77.243 us, total = 115.633 ms, Queueing time: mean = 107.665 us, max = 283.656 us, min = 12.278 us, total = 161.175 ms NodeManagerService.grpc_client.GetResourceLoad - 1497 total (0 active), Execution time: mean = 1.506 ms, total = 2.254 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 614 total (0 active), Execution time: mean = 40.924 us, total = 25.127 ms, Queueing time: mean = 102.247 us, max = 228.583 us, min = 11.443 us, total = 62.780 ms InternalKVGcsService.grpc_server.InternalKVPut - 614 total (0 active), Execution time: mean = 571.230 us, total = 350.736 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 600 total (0 active), Execution time: mean = 33.831 us, total = 20.298 ms, Queueing time: mean = 118.005 us, max = 233.188 us, min = 20.329 us, total = 70.803 ms NodeInfoGcsService.grpc_server.CheckAlive - 600 total (0 active), Execution time: mean = 592.546 us, total = 355.528 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 501 total (1 active), Execution time: mean = 9.757 us, total = 4.888 ms, Queueing time: mean = 75.919 us, max = 232.777 us, min = 19.812 us, total = 38.035 ms HealthCheck - 498 total (0 active), Execution time: mean = 7.620 us, total = 3.795 ms, Queueing time: mean = 103.551 us, max = 399.856 us, min = 19.476 us, total = 51.568 ms GcsInMemoryStore.Get - 369 total (0 active), Execution time: mean = 23.064 us, total = 8.511 ms, Queueing time: mean = 13.607 us, max = 64.163 us, min = 4.068 us, total = 5.021 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 368 total (0 active), Execution time: mean = 52.870 us, total = 19.456 ms, Queueing time: mean = 107.735 us, max = 269.544 us, min = 13.883 us, total = 39.646 ms InternalKVGcsService.grpc_server.InternalKVGet - 368 total (0 active), Execution time: mean = 609.145 us, total = 224.165 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 300 total (0 active), Execution time: mean = 91.020 us, total = 27.306 ms, Queueing time: mean = 115.399 us, max = 296.469 us, min = 19.828 us, total = 34.620 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 300 total (0 active), Execution time: mean = 698.502 us, total = 209.551 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 150 total (1 active), Execution time: mean = 1.406 ms, total = 210.849 ms, Queueing time: mean = 73.214 us, max = 315.299 us, min = -0.000 s, total = 10.982 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 25 total (1 active, 1 running), Execution time: mean = 1.561 ms, total = 39.028 ms, Queueing time: mean = 56.972 us, max = 108.649 us, min = 18.942 us, total = 1.424 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:26:53,845 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 69176 total (1 active) Queueing time: mean = 50.870 us, max = 7.565 ms, min = 2.216 us, total = 3.519 s Execution time: mean = 259.374 us, total = 17.942 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 31437 total (0 active), Execution time: mean = 537.683 us, total = 16.903 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 31437 total (0 active), Execution time: mean = 27.509 us, total = 864.788 ms, Queueing time: mean = 109.104 us, max = 7.565 ms, min = 5.193 us, total = 3.430 s event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 28.491 us, total = 170.948 ms, Queueing time: mean = 10.777 us, max = 4.983 ms, min = 2.216 us, total = 64.662 ms GcsTaskManager.GcJobSummary - 301 total (1 active), Execution time: mean = 11.788 us, total = 3.548 ms, Queueing time: mean = 80.851 us, max = 229.756 us, min = 27.203 us, total = 24.336 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:26:53,845 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7483 total (25 active) Queueing time: mean = 20.228 us, max = 333.684 us, min = 2.330 us, total = 151.365 ms Execution time: mean = 3.918 s, total = 29317.043 s Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 31.181 us, total = 187.083 ms, Queueing time: mean = 11.455 us, max = 291.078 us, min = 2.330 us, total = 68.730 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 392 total (0 active), Execution time: mean = 43.571 us, total = 17.080 ms, Queueing time: mean = 108.919 us, max = 333.684 us, min = 14.442 us, total = 42.696 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 392 total (24 active), Execution time: mean = 74.787 s, total = 29316.570 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 321 total (0 active), Execution time: mean = 115.252 us, total = 36.996 ms, Queueing time: mean = 112.327 us, max = 247.252 us, min = 23.656 us, total = 36.057 ms InternalPubSubGcsService.grpc_server.GcsPublish - 321 total (0 active), Execution time: mean = 657.528 us, total = 211.067 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 536.474 us, total = 3.219 ms, Queueing time: mean = 77.334 us, max = 153.556 us, min = 52.516 us, total = 464.003 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:26:53,845 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6026 total (0 active) Queueing time: mean = 10.696 us, max = 1.423 ms, min = 568.000 ns, total = 64.455 ms Execution time: mean = 27.810 us, total = 167.581 ms Event stats: event_loop_lag_probe - 6000 total (0 active), Execution time: mean = 27.672 us, total = 166.032 ms, Queueing time: mean = 10.429 us, max = 1.423 ms, min = 1.850 us, total = 62.575 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:27:53,845 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 311 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 333 - cumulative published bytes: 258838 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:27:53,846 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 19114 total (4 active) Queueing time: mean = 173.967 us, max = 279.303 ms, min = -0.000 s, total = 3.325 s Execution time: mean = 255.964 us, total = 4.892 s Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 23.999 us, total = 149.756 ms, Queueing time: mean = 11.335 us, max = 13.097 ms, min = 2.088 us, total = 70.733 ms GcsInMemoryStore.Put - 2540 total (0 active), Execution time: mean = 115.655 us, total = 293.763 ms, Queueing time: mean = 813.398 us, max = 278.545 ms, min = 2.310 us, total = 2.066 s RayletLoadPulled - 1560 total (1 active), Execution time: mean = 242.558 us, total = 378.390 ms, Queueing time: mean = 75.831 us, max = 460.894 us, min = 11.570 us, total = 118.296 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1557 total (0 active), Execution time: mean = 76.994 us, total = 119.880 ms, Queueing time: mean = 106.209 us, max = 283.656 us, min = 12.278 us, total = 165.367 ms NodeManagerService.grpc_client.GetResourceLoad - 1557 total (0 active), Execution time: mean = 1.502 ms, total = 2.338 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 636 total (0 active), Execution time: mean = 40.691 us, total = 25.880 ms, Queueing time: mean = 102.007 us, max = 228.583 us, min = 11.443 us, total = 64.876 ms InternalKVGcsService.grpc_server.InternalKVPut - 636 total (0 active), Execution time: mean = 571.324 us, total = 363.362 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 624 total (0 active), Execution time: mean = 33.773 us, total = 21.074 ms, Queueing time: mean = 117.343 us, max = 233.188 us, min = 20.329 us, total = 73.222 ms NodeInfoGcsService.grpc_server.CheckAlive - 624 total (0 active), Execution time: mean = 591.768 us, total = 369.263 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 521 total (1 active), Execution time: mean = 9.718 us, total = 5.063 ms, Queueing time: mean = 75.786 us, max = 232.777 us, min = 17.067 us, total = 39.484 ms HealthCheck - 518 total (0 active), Execution time: mean = 7.571 us, total = 3.922 ms, Queueing time: mean = 102.968 us, max = 399.856 us, min = 19.476 us, total = 53.338 ms GcsInMemoryStore.Get - 380 total (0 active), Execution time: mean = 22.868 us, total = 8.690 ms, Queueing time: mean = 13.610 us, max = 64.163 us, min = 4.068 us, total = 5.172 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 379 total (0 active), Execution time: mean = 52.680 us, total = 19.966 ms, Queueing time: mean = 107.247 us, max = 269.544 us, min = 13.883 us, total = 40.647 ms InternalKVGcsService.grpc_server.InternalKVGet - 379 total (0 active), Execution time: mean = 608.547 us, total = 230.639 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 311 total (0 active), Execution time: mean = 90.688 us, total = 28.204 ms, Queueing time: mean = 115.369 us, max = 296.469 us, min = 19.828 us, total = 35.880 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 311 total (0 active), Execution time: mean = 697.181 us, total = 216.823 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 156 total (1 active), Execution time: mean = 1.407 ms, total = 219.497 ms, Queueing time: mean = 72.746 us, max = 315.299 us, min = -0.000 s, total = 11.348 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 26 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 40.228 ms, Queueing time: mean = 55.358 us, max = 108.649 us, min = 15.007 us, total = 1.439 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:27:53,847 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 71948 total (1 active) Queueing time: mean = 50.814 us, max = 7.565 ms, min = 2.216 us, total = 3.656 s Execution time: mean = 258.949 us, total = 18.631 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 32697 total (0 active), Execution time: mean = 536.865 us, total = 17.554 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 32697 total (0 active), Execution time: mean = 27.416 us, total = 896.415 ms, Queueing time: mean = 109.000 us, max = 7.565 ms, min = 5.193 us, total = 3.564 s event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 28.351 us, total = 176.911 ms, Queueing time: mean = 10.700 us, max = 4.983 ms, min = 2.216 us, total = 66.768 ms GcsTaskManager.GcJobSummary - 313 total (1 active), Execution time: mean = 11.684 us, total = 3.657 ms, Queueing time: mean = 80.366 us, max = 229.756 us, min = 21.532 us, total = 25.155 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:27:53,847 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 7771 total (25 active) Queueing time: mean = 20.152 us, max = 333.684 us, min = 2.330 us, total = 156.603 ms Execution time: mean = 3.780 s, total = 29377.290 s Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 31.108 us, total = 194.114 ms, Queueing time: mean = 11.440 us, max = 291.078 us, min = 2.330 us, total = 71.385 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 404 total (0 active), Execution time: mean = 43.497 us, total = 17.573 ms, Queueing time: mean = 108.982 us, max = 333.684 us, min = 14.442 us, total = 44.029 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 404 total (24 active), Execution time: mean = 72.715 s, total = 29376.801 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 333 total (0 active), Execution time: mean = 114.525 us, total = 38.137 ms, Queueing time: mean = 112.036 us, max = 247.252 us, min = 23.656 us, total = 37.308 ms InternalPubSubGcsService.grpc_server.GcsPublish - 333 total (0 active), Execution time: mean = 656.236 us, total = 218.527 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 536.474 us, total = 3.219 ms, Queueing time: mean = 77.334 us, max = 153.556 us, min = 52.516 us, total = 464.003 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:27:53,847 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6266 total (0 active) Queueing time: mean = 10.620 us, max = 1.423 ms, min = 568.000 ns, total = 66.543 ms Execution time: mean = 27.659 us, total = 173.314 ms Event stats: event_loop_lag_probe - 6240 total (0 active), Execution time: mean = 27.526 us, total = 171.765 ms, Queueing time: mean = 10.363 us, max = 1.423 ms, min = 1.850 us, total = 64.664 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:28:53,847 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 323 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 347 - cumulative published bytes: 268734 - current buffered bytes: 153 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:28:53,848 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 19833 total (4 active) Queueing time: mean = 169.280 us, max = 279.303 ms, min = -0.000 s, total = 3.357 s Execution time: mean = 255.827 us, total = 5.074 s Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 24.116 us, total = 156.270 ms, Queueing time: mean = 11.306 us, max = 13.097 ms, min = 2.088 us, total = 73.265 ms GcsInMemoryStore.Put - 2636 total (0 active), Execution time: mean = 111.664 us, total = 294.346 ms, Queueing time: mean = 785.468 us, max = 278.545 ms, min = 2.310 us, total = 2.070 s RayletLoadPulled - 1620 total (1 active), Execution time: mean = 243.448 us, total = 394.386 ms, Queueing time: mean = 76.142 us, max = 460.894 us, min = 11.570 us, total = 123.350 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1617 total (0 active), Execution time: mean = 77.109 us, total = 124.685 ms, Queueing time: mean = 106.144 us, max = 283.656 us, min = 12.278 us, total = 171.634 ms NodeManagerService.grpc_client.GetResourceLoad - 1617 total (0 active), Execution time: mean = 1.503 ms, total = 2.430 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 660 total (0 active), Execution time: mean = 40.896 us, total = 26.991 ms, Queueing time: mean = 102.359 us, max = 228.583 us, min = 11.443 us, total = 67.557 ms InternalKVGcsService.grpc_server.InternalKVPut - 660 total (0 active), Execution time: mean = 571.747 us, total = 377.353 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 648 total (0 active), Execution time: mean = 33.936 us, total = 21.990 ms, Queueing time: mean = 119.311 us, max = 1.498 ms, min = 20.329 us, total = 77.314 ms NodeInfoGcsService.grpc_server.CheckAlive - 648 total (0 active), Execution time: mean = 594.260 us, total = 385.080 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 541 total (1 active), Execution time: mean = 9.723 us, total = 5.260 ms, Queueing time: mean = 75.871 us, max = 232.777 us, min = 17.067 us, total = 41.046 ms HealthCheck - 538 total (0 active), Execution time: mean = 7.567 us, total = 4.071 ms, Queueing time: mean = 103.311 us, max = 399.856 us, min = 19.476 us, total = 55.582 ms GcsInMemoryStore.Get - 392 total (0 active), Execution time: mean = 22.944 us, total = 8.994 ms, Queueing time: mean = 13.578 us, max = 64.163 us, min = 4.068 us, total = 5.322 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 391 total (0 active), Execution time: mean = 52.817 us, total = 20.652 ms, Queueing time: mean = 107.232 us, max = 269.544 us, min = 13.883 us, total = 41.928 ms InternalKVGcsService.grpc_server.InternalKVGet - 391 total (0 active), Execution time: mean = 608.880 us, total = 238.072 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 323 total (0 active), Execution time: mean = 91.819 us, total = 29.658 ms, Queueing time: mean = 115.188 us, max = 296.469 us, min = 19.828 us, total = 37.206 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 323 total (0 active), Execution time: mean = 698.597 us, total = 225.647 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 162 total (1 active), Execution time: mean = 1.408 ms, total = 228.135 ms, Queueing time: mean = 72.470 us, max = 315.299 us, min = -0.000 s, total = 11.740 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 27 total (1 active, 1 running), Execution time: mean = 1.555 ms, total = 41.978 ms, Queueing time: mean = 55.969 us, max = 108.649 us, min = 15.007 us, total = 1.511 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:28:53,848 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 74720 total (1 active) Queueing time: mean = 50.784 us, max = 7.565 ms, min = 2.216 us, total = 3.795 s Execution time: mean = 258.697 us, total = 19.330 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 33957 total (0 active), Execution time: mean = 536.306 us, total = 18.211 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 33957 total (0 active), Execution time: mean = 27.422 us, total = 931.167 ms, Queueing time: mean = 108.936 us, max = 7.565 ms, min = 3.372 us, total = 3.699 s event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 28.316 us, total = 183.488 ms, Queueing time: mean = 10.670 us, max = 4.983 ms, min = 2.216 us, total = 69.141 ms GcsTaskManager.GcJobSummary - 325 total (1 active), Execution time: mean = 11.694 us, total = 3.800 ms, Queueing time: mean = 80.527 us, max = 229.756 us, min = 21.532 us, total = 26.171 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:28:53,849 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8067 total (25 active) Queueing time: mean = 20.092 us, max = 333.684 us, min = 2.330 us, total = 162.085 ms Execution time: mean = 3.649 s, total = 29437.366 s Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 31.079 us, total = 201.390 ms, Queueing time: mean = 11.446 us, max = 291.078 us, min = 2.330 us, total = 74.170 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 418 total (0 active), Execution time: mean = 43.463 us, total = 18.168 ms, Queueing time: mean = 108.702 us, max = 333.684 us, min = 14.442 us, total = 45.438 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 418 total (24 active), Execution time: mean = 70.423 s, total = 29436.860 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 347 total (0 active), Execution time: mean = 113.812 us, total = 39.493 ms, Queueing time: mean = 111.228 us, max = 247.252 us, min = 23.656 us, total = 38.596 ms InternalPubSubGcsService.grpc_server.GcsPublish - 347 total (0 active), Execution time: mean = 654.578 us, total = 227.139 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 536.474 us, total = 3.219 ms, Queueing time: mean = 77.334 us, max = 153.556 us, min = 52.516 us, total = 464.003 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:28:53,849 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6506 total (0 active) Queueing time: mean = 10.589 us, max = 1.423 ms, min = 568.000 ns, total = 68.893 ms Execution time: mean = 27.648 us, total = 179.877 ms Event stats: event_loop_lag_probe - 6480 total (0 active), Execution time: mean = 27.520 us, total = 178.328 ms, Queueing time: mean = 10.342 us, max = 1.423 ms, min = 1.850 us, total = 67.014 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:29:53,849 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 335 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 360 - cumulative published bytes: 278596 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:29:53,850 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 20552 total (4 active) Queueing time: mean = 164.932 us, max = 279.303 ms, min = -0.000 s, total = 3.390 s Execution time: mean = 255.853 us, total = 5.258 s Event stats: event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 24.180 us, total = 162.488 ms, Queueing time: mean = 11.247 us, max = 13.097 ms, min = 2.088 us, total = 75.579 ms GcsInMemoryStore.Put - 2732 total (0 active), Execution time: mean = 107.946 us, total = 294.907 ms, Queueing time: mean = 759.618 us, max = 278.545 ms, min = 2.310 us, total = 2.075 s RayletLoadPulled - 1680 total (1 active), Execution time: mean = 244.313 us, total = 410.446 ms, Queueing time: mean = 76.360 us, max = 460.894 us, min = 11.570 us, total = 128.286 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1677 total (0 active), Execution time: mean = 77.379 us, total = 129.765 ms, Queueing time: mean = 106.689 us, max = 330.146 us, min = 12.278 us, total = 178.917 ms NodeManagerService.grpc_client.GetResourceLoad - 1677 total (0 active), Execution time: mean = 1.506 ms, total = 2.526 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 684 total (0 active), Execution time: mean = 41.104 us, total = 28.115 ms, Queueing time: mean = 102.625 us, max = 228.583 us, min = 11.443 us, total = 70.195 ms InternalKVGcsService.grpc_server.InternalKVPut - 684 total (0 active), Execution time: mean = 572.029 us, total = 391.268 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 672 total (0 active), Execution time: mean = 34.068 us, total = 22.894 ms, Queueing time: mean = 119.790 us, max = 1.498 ms, min = 20.329 us, total = 80.499 ms NodeInfoGcsService.grpc_server.CheckAlive - 672 total (0 active), Execution time: mean = 594.931 us, total = 399.794 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 561 total (1 active), Execution time: mean = 9.725 us, total = 5.456 ms, Queueing time: mean = 76.257 us, max = 232.777 us, min = 17.067 us, total = 42.780 ms HealthCheck - 558 total (0 active), Execution time: mean = 7.574 us, total = 4.226 ms, Queueing time: mean = 103.292 us, max = 399.856 us, min = 19.476 us, total = 57.637 ms GcsInMemoryStore.Get - 404 total (0 active), Execution time: mean = 22.838 us, total = 9.226 ms, Queueing time: mean = 13.557 us, max = 64.163 us, min = 4.068 us, total = 5.477 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 403 total (0 active), Execution time: mean = 52.964 us, total = 21.345 ms, Queueing time: mean = 107.219 us, max = 269.544 us, min = 13.883 us, total = 43.209 ms InternalKVGcsService.grpc_server.InternalKVGet - 403 total (0 active), Execution time: mean = 608.661 us, total = 245.291 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 335 total (0 active), Execution time: mean = 92.685 us, total = 31.049 ms, Queueing time: mean = 115.497 us, max = 296.469 us, min = 19.828 us, total = 38.691 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 335 total (0 active), Execution time: mean = 700.706 us, total = 234.737 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 168 total (1 active), Execution time: mean = 1.413 ms, total = 237.322 ms, Queueing time: mean = 72.520 us, max = 315.299 us, min = -0.000 s, total = 12.183 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 28 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 43.635 ms, Queueing time: mean = 55.839 us, max = 108.649 us, min = 15.007 us, total = 1.563 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:29:53,850 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 77491 total (1 active) Queueing time: mean = 50.968 us, max = 7.565 ms, min = 2.216 us, total = 3.950 s Execution time: mean = 259.035 us, total = 20.073 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 35217 total (0 active), Execution time: mean = 536.864 us, total = 18.907 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 35217 total (0 active), Execution time: mean = 27.586 us, total = 971.481 ms, Queueing time: mean = 109.333 us, max = 7.565 ms, min = 3.372 us, total = 3.850 s event_loop_lag_probe - 6719 total (0 active), Execution time: mean = 28.375 us, total = 190.654 ms, Queueing time: mean = 10.659 us, max = 4.983 ms, min = 2.216 us, total = 71.619 ms GcsTaskManager.GcJobSummary - 337 total (1 active), Execution time: mean = 11.709 us, total = 3.946 ms, Queueing time: mean = 81.391 us, max = 229.756 us, min = 21.532 us, total = 27.429 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:29:53,850 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8358 total (25 active) Queueing time: mean = 20.101 us, max = 333.684 us, min = 2.330 us, total = 168.002 ms Execution time: mean = 3.529 s, total = 29497.580 s Event stats: event_loop_lag_probe - 6719 total (0 active), Execution time: mean = 31.161 us, total = 209.372 ms, Queueing time: mean = 11.460 us, max = 291.078 us, min = 2.330 us, total = 77.001 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 431 total (0 active), Execution time: mean = 43.510 us, total = 18.753 ms, Queueing time: mean = 108.906 us, max = 333.684 us, min = 14.442 us, total = 46.939 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 431 total (24 active), Execution time: mean = 68.439 s, total = 29497.053 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 360 total (0 active), Execution time: mean = 114.808 us, total = 41.331 ms, Queueing time: mean = 111.615 us, max = 247.252 us, min = 23.656 us, total = 40.181 ms InternalPubSubGcsService.grpc_server.GcsPublish - 360 total (0 active), Execution time: mean = 657.295 us, total = 236.626 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 536.474 us, total = 3.219 ms, Queueing time: mean = 77.334 us, max = 153.556 us, min = 52.516 us, total = 464.003 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:29:53,850 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6746 total (0 active) Queueing time: mean = 10.559 us, max = 1.423 ms, min = 568.000 ns, total = 71.232 ms Execution time: mean = 27.617 us, total = 186.304 ms Event stats: event_loop_lag_probe - 6720 total (0 active), Execution time: mean = 27.493 us, total = 184.755 ms, Queueing time: mean = 10.320 us, max = 1.423 ms, min = 1.850 us, total = 69.352 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:30:53,850 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 347 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 372 - cumulative published bytes: 288424 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:30:53,851 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 21271 total (4 active) Queueing time: mean = 160.839 us, max = 279.303 ms, min = -0.000 s, total = 3.421 s Execution time: mean = 255.799 us, total = 5.441 s Event stats: event_loop_lag_probe - 6960 total (0 active), Execution time: mean = 24.163 us, total = 168.177 ms, Queueing time: mean = 11.169 us, max = 13.097 ms, min = 2.088 us, total = 77.738 ms GcsInMemoryStore.Put - 2828 total (0 active), Execution time: mean = 104.455 us, total = 295.399 ms, Queueing time: mean = 735.377 us, max = 278.545 ms, min = 2.310 us, total = 2.080 s RayletLoadPulled - 1740 total (1 active), Execution time: mean = 244.525 us, total = 425.473 ms, Queueing time: mean = 76.484 us, max = 460.894 us, min = 11.570 us, total = 133.082 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1737 total (0 active), Execution time: mean = 77.365 us, total = 134.383 ms, Queueing time: mean = 107.242 us, max = 330.146 us, min = 12.278 us, total = 186.280 ms NodeManagerService.grpc_client.GetResourceLoad - 1737 total (0 active), Execution time: mean = 1.510 ms, total = 2.623 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 708 total (0 active), Execution time: mean = 41.167 us, total = 29.146 ms, Queueing time: mean = 103.473 us, max = 228.583 us, min = 11.443 us, total = 73.259 ms InternalKVGcsService.grpc_server.InternalKVPut - 708 total (0 active), Execution time: mean = 572.990 us, total = 405.677 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 696 total (0 active), Execution time: mean = 34.119 us, total = 23.747 ms, Queueing time: mean = 119.479 us, max = 1.498 ms, min = 20.329 us, total = 83.158 ms NodeInfoGcsService.grpc_server.CheckAlive - 696 total (0 active), Execution time: mean = 595.752 us, total = 414.644 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 581 total (1 active), Execution time: mean = 9.693 us, total = 5.631 ms, Queueing time: mean = 76.294 us, max = 232.777 us, min = 17.067 us, total = 44.327 ms HealthCheck - 578 total (0 active), Execution time: mean = 7.575 us, total = 4.379 ms, Queueing time: mean = 102.627 us, max = 399.856 us, min = 19.476 us, total = 59.318 ms GcsInMemoryStore.Get - 416 total (0 active), Execution time: mean = 22.775 us, total = 9.474 ms, Queueing time: mean = 13.626 us, max = 64.163 us, min = 4.068 us, total = 5.668 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 415 total (0 active), Execution time: mean = 53.054 us, total = 22.017 ms, Queueing time: mean = 107.922 us, max = 269.544 us, min = 13.883 us, total = 44.788 ms InternalKVGcsService.grpc_server.InternalKVGet - 415 total (0 active), Execution time: mean = 608.921 us, total = 252.702 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 347 total (0 active), Execution time: mean = 92.822 us, total = 32.209 ms, Queueing time: mean = 116.043 us, max = 296.469 us, min = 19.828 us, total = 40.267 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 347 total (0 active), Execution time: mean = 700.088 us, total = 242.931 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 174 total (1 active), Execution time: mean = 1.421 ms, total = 247.236 ms, Queueing time: mean = 72.768 us, max = 315.299 us, min = -0.000 s, total = 12.662 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 29 total (1 active, 1 running), Execution time: mean = 1.546 ms, total = 44.835 ms, Queueing time: mean = 55.936 us, max = 108.649 us, min = 15.007 us, total = 1.622 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:30:53,851 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 80263 total (1 active) Queueing time: mean = 50.961 us, max = 7.565 ms, min = 2.216 us, total = 4.090 s Execution time: mean = 259.045 us, total = 20.792 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 36477 total (0 active), Execution time: mean = 536.873 us, total = 19.584 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 36477 total (0 active), Execution time: mean = 27.592 us, total = 1.006 s, Queueing time: mean = 109.309 us, max = 7.565 ms, min = 3.372 us, total = 3.987 s event_loop_lag_probe - 6959 total (0 active), Execution time: mean = 28.395 us, total = 197.601 ms, Queueing time: mean = 10.650 us, max = 4.983 ms, min = 2.216 us, total = 74.110 ms GcsTaskManager.GcJobSummary - 349 total (1 active), Execution time: mean = 11.715 us, total = 4.089 ms, Queueing time: mean = 82.635 us, max = 229.756 us, min = 21.532 us, total = 28.840 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:30:53,851 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8646 total (25 active) Queueing time: mean = 20.108 us, max = 333.684 us, min = 2.330 us, total = 173.850 ms Execution time: mean = 3.419 s, total = 29557.765 s Event stats: event_loop_lag_probe - 6959 total (0 active), Execution time: mean = 31.173 us, total = 216.934 ms, Queueing time: mean = 11.465 us, max = 291.078 us, min = 2.330 us, total = 79.783 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 443 total (0 active), Execution time: mean = 43.829 us, total = 19.416 ms, Queueing time: mean = 109.281 us, max = 333.684 us, min = 14.442 us, total = 48.412 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 443 total (24 active), Execution time: mean = 66.721 s, total = 29557.221 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 372 total (0 active), Execution time: mean = 115.375 us, total = 42.920 ms, Queueing time: mean = 112.295 us, max = 247.252 us, min = 23.656 us, total = 41.774 ms InternalPubSubGcsService.grpc_server.GcsPublish - 372 total (0 active), Execution time: mean = 658.119 us, total = 244.820 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 6 total (1 active), Execution time: mean = 536.474 us, total = 3.219 ms, Queueing time: mean = 77.334 us, max = 153.556 us, min = 52.516 us, total = 464.003 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:30:53,852 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 6986 total (0 active) Queueing time: mean = 10.537 us, max = 1.423 ms, min = 568.000 ns, total = 73.610 ms Execution time: mean = 27.584 us, total = 192.703 ms Event stats: event_loop_lag_probe - 6960 total (0 active), Execution time: mean = 27.465 us, total = 191.154 ms, Queueing time: mean = 10.306 us, max = 1.423 ms, min = 1.850 us, total = 71.731 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:31:53,852 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 359 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 384 - cumulative published bytes: 298252 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:31:53,853 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 21989 total (4 active) Queueing time: mean = 156.920 us, max = 279.303 ms, min = -0.000 s, total = 3.451 s Execution time: mean = 255.492 us, total = 5.618 s Event stats: event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 24.161 us, total = 173.938 ms, Queueing time: mean = 11.119 us, max = 13.097 ms, min = 2.088 us, total = 80.047 ms GcsInMemoryStore.Put - 2924 total (0 active), Execution time: mean = 101.183 us, total = 295.859 ms, Queueing time: mean = 712.880 us, max = 278.545 ms, min = 2.310 us, total = 2.084 s RayletLoadPulled - 1800 total (1 active), Execution time: mean = 244.772 us, total = 440.589 ms, Queueing time: mean = 76.121 us, max = 460.894 us, min = 11.570 us, total = 137.018 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1797 total (0 active), Execution time: mean = 77.381 us, total = 139.054 ms, Queueing time: mean = 107.108 us, max = 330.146 us, min = 12.278 us, total = 192.473 ms NodeManagerService.grpc_client.GetResourceLoad - 1797 total (0 active), Execution time: mean = 1.512 ms, total = 2.716 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 732 total (0 active), Execution time: mean = 41.242 us, total = 30.189 ms, Queueing time: mean = 103.452 us, max = 228.583 us, min = 11.443 us, total = 75.727 ms InternalKVGcsService.grpc_server.InternalKVPut - 732 total (0 active), Execution time: mean = 573.233 us, total = 419.607 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 720 total (0 active), Execution time: mean = 34.060 us, total = 24.523 ms, Queueing time: mean = 119.693 us, max = 1.498 ms, min = 20.329 us, total = 86.179 ms NodeInfoGcsService.grpc_server.CheckAlive - 720 total (0 active), Execution time: mean = 596.760 us, total = 429.668 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 601 total (1 active), Execution time: mean = 9.704 us, total = 5.832 ms, Queueing time: mean = 76.062 us, max = 232.777 us, min = 17.067 us, total = 45.713 ms HealthCheck - 598 total (0 active), Execution time: mean = 7.596 us, total = 4.542 ms, Queueing time: mean = 102.078 us, max = 399.856 us, min = 19.476 us, total = 61.043 ms GcsInMemoryStore.Get - 428 total (0 active), Execution time: mean = 22.637 us, total = 9.689 ms, Queueing time: mean = 13.639 us, max = 64.163 us, min = 4.068 us, total = 5.838 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 427 total (0 active), Execution time: mean = 53.040 us, total = 22.648 ms, Queueing time: mean = 108.166 us, max = 269.544 us, min = 13.883 us, total = 46.187 ms InternalKVGcsService.grpc_server.InternalKVGet - 427 total (0 active), Execution time: mean = 608.772 us, total = 259.946 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 359 total (0 active), Execution time: mean = 92.592 us, total = 33.241 ms, Queueing time: mean = 116.195 us, max = 296.469 us, min = 19.828 us, total = 41.714 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 359 total (0 active), Execution time: mean = 699.491 us, total = 251.117 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 180 total (1 active), Execution time: mean = 1.415 ms, total = 254.748 ms, Queueing time: mean = 72.301 us, max = 315.299 us, min = -0.000 s, total = 13.014 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 30 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 46.397 ms, Queueing time: mean = 57.185 us, max = 108.649 us, min = 15.007 us, total = 1.716 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:31:53,853 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 83035 total (1 active) Queueing time: mean = 50.977 us, max = 7.565 ms, min = 2.216 us, total = 4.233 s Execution time: mean = 259.053 us, total = 21.510 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 37737 total (0 active), Execution time: mean = 536.872 us, total = 20.260 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 37737 total (0 active), Execution time: mean = 27.583 us, total = 1.041 s, Queueing time: mean = 109.338 us, max = 7.565 ms, min = 3.372 us, total = 4.126 s event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 28.527 us, total = 205.363 ms, Queueing time: mean = 10.663 us, max = 4.983 ms, min = 2.216 us, total = 76.760 ms GcsTaskManager.GcJobSummary - 361 total (1 active), Execution time: mean = 11.676 us, total = 4.215 ms, Queueing time: mean = 83.013 us, max = 229.756 us, min = 21.532 us, total = 29.968 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:31:53,853 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 8979 total (25 active) Queueing time: mean = 20.247 us, max = 333.684 us, min = 2.330 us, total = 181.798 ms Execution time: mean = 4.769 s, total = 42817.941 s Event stats: event_loop_lag_probe - 7199 total (0 active), Execution time: mean = 31.207 us, total = 224.660 ms, Queueing time: mean = 11.475 us, max = 291.078 us, min = 2.330 us, total = 82.611 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 477 total (0 active), Execution time: mean = 42.811 us, total = 20.421 ms, Queueing time: mean = 108.938 us, max = 333.684 us, min = 14.442 us, total = 51.963 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 477 total (24 active), Execution time: mean = 89.764 s, total = 42817.377 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 384 total (0 active), Execution time: mean = 115.634 us, total = 44.404 ms, Queueing time: mean = 112.541 us, max = 247.252 us, min = 23.656 us, total = 43.216 ms InternalPubSubGcsService.grpc_server.GcsPublish - 384 total (0 active), Execution time: mean = 658.174 us, total = 252.739 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 604.476 us, total = 4.231 ms, Queueing time: mean = 84.400 us, max = 153.556 us, min = 52.516 us, total = 590.801 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:31:53,853 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7226 total (0 active) Queueing time: mean = 10.516 us, max = 1.423 ms, min = 568.000 ns, total = 75.990 ms Execution time: mean = 27.577 us, total = 199.271 ms Event stats: event_loop_lag_probe - 7200 total (0 active), Execution time: mean = 27.461 us, total = 197.722 ms, Queueing time: mean = 10.293 us, max = 1.423 ms, min = 1.850 us, total = 74.111 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:32:53,854 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 371 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 396 - cumulative published bytes: 308080 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:32:53,855 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 22708 total (4 active) Queueing time: mean = 153.156 us, max = 279.303 ms, min = -0.000 s, total = 3.478 s Execution time: mean = 254.987 us, total = 5.790 s Event stats: event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 24.174 us, total = 179.827 ms, Queueing time: mean = 11.051 us, max = 13.097 ms, min = 2.088 us, total = 82.211 ms GcsInMemoryStore.Put - 3020 total (0 active), Execution time: mean = 98.117 us, total = 296.312 ms, Queueing time: mean = 691.673 us, max = 278.545 ms, min = 2.310 us, total = 2.089 s RayletLoadPulled - 1860 total (1 active), Execution time: mean = 245.056 us, total = 455.804 ms, Queueing time: mean = 75.732 us, max = 460.894 us, min = 11.570 us, total = 140.862 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1857 total (0 active), Execution time: mean = 77.358 us, total = 143.654 ms, Queueing time: mean = 106.574 us, max = 330.146 us, min = 12.278 us, total = 197.908 ms NodeManagerService.grpc_client.GetResourceLoad - 1857 total (0 active), Execution time: mean = 1.510 ms, total = 2.805 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 756 total (0 active), Execution time: mean = 41.135 us, total = 31.098 ms, Queueing time: mean = 103.317 us, max = 228.583 us, min = 11.443 us, total = 78.108 ms InternalKVGcsService.grpc_server.InternalKVPut - 756 total (0 active), Execution time: mean = 572.867 us, total = 433.087 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 744 total (0 active), Execution time: mean = 34.068 us, total = 25.346 ms, Queueing time: mean = 119.642 us, max = 1.498 ms, min = 20.329 us, total = 89.014 ms NodeInfoGcsService.grpc_server.CheckAlive - 744 total (0 active), Execution time: mean = 597.264 us, total = 444.364 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 621 total (1 active), Execution time: mean = 9.676 us, total = 6.009 ms, Queueing time: mean = 76.102 us, max = 232.777 us, min = 17.067 us, total = 47.259 ms HealthCheck - 618 total (0 active), Execution time: mean = 7.643 us, total = 4.724 ms, Queueing time: mean = 101.760 us, max = 399.856 us, min = 19.476 us, total = 62.888 ms GcsInMemoryStore.Get - 440 total (0 active), Execution time: mean = 22.526 us, total = 9.912 ms, Queueing time: mean = 13.608 us, max = 64.163 us, min = 4.068 us, total = 5.988 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 439 total (0 active), Execution time: mean = 53.334 us, total = 23.413 ms, Queueing time: mean = 107.897 us, max = 269.544 us, min = 13.883 us, total = 47.367 ms InternalKVGcsService.grpc_server.InternalKVGet - 439 total (0 active), Execution time: mean = 608.555 us, total = 267.156 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 371 total (0 active), Execution time: mean = 92.640 us, total = 34.369 ms, Queueing time: mean = 115.543 us, max = 296.469 us, min = 19.828 us, total = 42.866 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 371 total (0 active), Execution time: mean = 698.860 us, total = 259.277 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 186 total (1 active), Execution time: mean = 1.414 ms, total = 262.947 ms, Queueing time: mean = 71.989 us, max = 315.299 us, min = -0.000 s, total = 13.390 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 31 total (1 active, 1 running), Execution time: mean = 1.551 ms, total = 48.081 ms, Queueing time: mean = 57.347 us, max = 108.649 us, min = 15.007 us, total = 1.778 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:32:53,855 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 85807 total (1 active) Queueing time: mean = 51.023 us, max = 7.565 ms, min = 2.216 us, total = 4.378 s Execution time: mean = 259.213 us, total = 22.242 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 38997 total (0 active), Execution time: mean = 537.256 us, total = 20.951 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 38997 total (0 active), Execution time: mean = 27.561 us, total = 1.075 s, Queueing time: mean = 109.449 us, max = 7.565 ms, min = 3.372 us, total = 4.268 s event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 28.462 us, total = 211.731 ms, Queueing time: mean = 10.617 us, max = 4.983 ms, min = 2.216 us, total = 78.983 ms GcsTaskManager.GcJobSummary - 373 total (1 active), Execution time: mean = 11.697 us, total = 4.363 ms, Queueing time: mean = 82.717 us, max = 229.756 us, min = 21.532 us, total = 30.853 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:32:53,855 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9267 total (25 active) Queueing time: mean = 20.196 us, max = 333.684 us, min = 2.330 us, total = 187.160 ms Execution time: mean = 4.627 s, total = 42878.136 s Event stats: event_loop_lag_probe - 7439 total (0 active), Execution time: mean = 31.214 us, total = 232.200 ms, Queueing time: mean = 11.469 us, max = 291.078 us, min = 2.330 us, total = 85.321 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 489 total (0 active), Execution time: mean = 42.931 us, total = 20.993 ms, Queueing time: mean = 109.137 us, max = 333.684 us, min = 14.442 us, total = 53.368 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 489 total (24 active), Execution time: mean = 87.684 s, total = 42877.555 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 396 total (0 active), Execution time: mean = 115.100 us, total = 45.580 ms, Queueing time: mean = 112.281 us, max = 247.252 us, min = 23.656 us, total = 44.463 ms InternalPubSubGcsService.grpc_server.GcsPublish - 396 total (0 active), Execution time: mean = 658.564 us, total = 260.791 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 604.476 us, total = 4.231 ms, Queueing time: mean = 84.400 us, max = 153.556 us, min = 52.516 us, total = 590.801 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:32:53,855 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7466 total (0 active) Queueing time: mean = 10.513 us, max = 1.423 ms, min = 568.000 ns, total = 78.493 ms Execution time: mean = 27.596 us, total = 206.032 ms Event stats: event_loop_lag_probe - 7440 total (0 active), Execution time: mean = 27.484 us, total = 204.483 ms, Queueing time: mean = 10.298 us, max = 1.423 ms, min = 1.850 us, total = 76.614 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:33:53,855 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 383 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 408 - cumulative published bytes: 317908 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:33:53,856 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 23427 total (4 active) Queueing time: mean = 149.825 us, max = 279.303 ms, min = -0.000 s, total = 3.510 s Execution time: mean = 254.963 us, total = 5.973 s Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 24.215 us, total = 185.947 ms, Queueing time: mean = 11.001 us, max = 13.097 ms, min = 2.088 us, total = 84.476 ms GcsInMemoryStore.Put - 3116 total (0 active), Execution time: mean = 95.258 us, total = 296.823 ms, Queueing time: mean = 671.895 us, max = 278.545 ms, min = 2.310 us, total = 2.094 s RayletLoadPulled - 1920 total (1 active), Execution time: mean = 245.754 us, total = 471.848 ms, Queueing time: mean = 75.858 us, max = 460.894 us, min = 11.570 us, total = 145.647 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1917 total (0 active), Execution time: mean = 77.477 us, total = 148.523 ms, Queueing time: mean = 106.921 us, max = 330.146 us, min = 12.278 us, total = 204.967 ms NodeManagerService.grpc_client.GetResourceLoad - 1917 total (0 active), Execution time: mean = 1.512 ms, total = 2.899 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 780 total (0 active), Execution time: mean = 41.453 us, total = 32.333 ms, Queueing time: mean = 103.469 us, max = 228.583 us, min = 11.443 us, total = 80.706 ms InternalKVGcsService.grpc_server.InternalKVPut - 780 total (0 active), Execution time: mean = 573.409 us, total = 447.259 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 768 total (0 active), Execution time: mean = 34.184 us, total = 26.253 ms, Queueing time: mean = 120.077 us, max = 1.498 ms, min = 20.329 us, total = 92.219 ms NodeInfoGcsService.grpc_server.CheckAlive - 768 total (0 active), Execution time: mean = 598.711 us, total = 459.810 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 641 total (1 active), Execution time: mean = 9.668 us, total = 6.197 ms, Queueing time: mean = 76.286 us, max = 232.777 us, min = 17.067 us, total = 48.900 ms HealthCheck - 638 total (0 active), Execution time: mean = 7.653 us, total = 4.883 ms, Queueing time: mean = 101.833 us, max = 399.856 us, min = 19.476 us, total = 64.969 ms GcsInMemoryStore.Get - 452 total (0 active), Execution time: mean = 22.434 us, total = 10.140 ms, Queueing time: mean = 13.633 us, max = 64.163 us, min = 4.068 us, total = 6.162 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 451 total (0 active), Execution time: mean = 53.557 us, total = 24.154 ms, Queueing time: mean = 108.120 us, max = 269.544 us, min = 13.883 us, total = 48.762 ms InternalKVGcsService.grpc_server.InternalKVGet - 451 total (0 active), Execution time: mean = 609.057 us, total = 274.685 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 383 total (0 active), Execution time: mean = 93.218 us, total = 35.702 ms, Queueing time: mean = 116.022 us, max = 296.469 us, min = 19.828 us, total = 44.436 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 383 total (0 active), Execution time: mean = 699.005 us, total = 267.719 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 192 total (1 active), Execution time: mean = 1.418 ms, total = 272.194 ms, Queueing time: mean = 72.195 us, max = 315.299 us, min = -0.000 s, total = 13.861 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 32 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 49.870 ms, Queueing time: mean = 57.661 us, max = 108.649 us, min = 15.007 us, total = 1.845 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:33:53,857 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 88579 total (1 active) Queueing time: mean = 51.048 us, max = 7.565 ms, min = 2.216 us, total = 4.522 s Execution time: mean = 259.335 us, total = 22.972 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 40257 total (0 active), Execution time: mean = 537.484 us, total = 21.637 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 40257 total (0 active), Execution time: mean = 27.598 us, total = 1.111 s, Queueing time: mean = 109.509 us, max = 7.565 ms, min = 3.372 us, total = 4.409 s event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 28.469 us, total = 218.616 ms, Queueing time: mean = 10.600 us, max = 4.983 ms, min = 2.216 us, total = 81.396 ms GcsTaskManager.GcJobSummary - 385 total (1 active), Execution time: mean = 11.674 us, total = 4.494 ms, Queueing time: mean = 82.579 us, max = 229.756 us, min = 21.532 us, total = 31.793 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:33:53,857 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9555 total (25 active) Queueing time: mean = 20.163 us, max = 333.684 us, min = 2.330 us, total = 192.656 ms Execution time: mean = 4.494 s, total = 42938.257 s Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 31.256 us, total = 240.016 ms, Queueing time: mean = 11.491 us, max = 291.078 us, min = 2.330 us, total = 88.237 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 501 total (0 active), Execution time: mean = 43.057 us, total = 21.571 ms, Queueing time: mean = 108.977 us, max = 333.684 us, min = 14.442 us, total = 54.598 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 501 total (24 active), Execution time: mean = 85.704 s, total = 42937.659 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 408 total (0 active), Execution time: mean = 115.252 us, total = 47.023 ms, Queueing time: mean = 112.286 us, max = 247.252 us, min = 23.656 us, total = 45.813 ms InternalPubSubGcsService.grpc_server.GcsPublish - 408 total (0 active), Execution time: mean = 658.932 us, total = 268.844 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 604.476 us, total = 4.231 ms, Queueing time: mean = 84.400 us, max = 153.556 us, min = 52.516 us, total = 590.801 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:33:53,857 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7705 total (0 active) Queueing time: mean = 10.531 us, max = 1.423 ms, min = 568.000 ns, total = 81.145 ms Execution time: mean = 27.671 us, total = 213.205 ms Event stats: event_loop_lag_probe - 7679 total (0 active), Execution time: mean = 27.563 us, total = 211.656 ms, Queueing time: mean = 10.322 us, max = 1.423 ms, min = 1.850 us, total = 79.265 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:34:53,857 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 395 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 420 - cumulative published bytes: 327736 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:34:53,858 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 24146 total (4 active) Queueing time: mean = 146.556 us, max = 279.303 ms, min = -0.000 s, total = 3.539 s Execution time: mean = 254.451 us, total = 6.144 s Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 24.180 us, total = 191.480 ms, Queueing time: mean = 10.930 us, max = 13.097 ms, min = 2.088 us, total = 86.557 ms GcsInMemoryStore.Put - 3212 total (0 active), Execution time: mean = 92.572 us, total = 297.343 ms, Queueing time: mean = 653.291 us, max = 278.545 ms, min = 2.310 us, total = 2.098 s RayletLoadPulled - 1980 total (1 active), Execution time: mean = 246.021 us, total = 487.122 ms, Queueing time: mean = 75.792 us, max = 460.894 us, min = 11.570 us, total = 150.068 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 1977 total (0 active), Execution time: mean = 77.439 us, total = 153.097 ms, Queueing time: mean = 106.748 us, max = 330.146 us, min = 12.278 us, total = 211.041 ms NodeManagerService.grpc_client.GetResourceLoad - 1977 total (0 active), Execution time: mean = 1.511 ms, total = 2.988 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 804 total (0 active), Execution time: mean = 41.249 us, total = 33.164 ms, Queueing time: mean = 103.282 us, max = 228.583 us, min = 11.443 us, total = 83.039 ms InternalKVGcsService.grpc_server.InternalKVPut - 804 total (0 active), Execution time: mean = 572.433 us, total = 460.236 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 792 total (0 active), Execution time: mean = 34.114 us, total = 27.018 ms, Queueing time: mean = 119.561 us, max = 1.498 ms, min = 20.329 us, total = 94.692 ms NodeInfoGcsService.grpc_server.CheckAlive - 792 total (0 active), Execution time: mean = 597.903 us, total = 473.539 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 661 total (1 active), Execution time: mean = 9.676 us, total = 6.396 ms, Queueing time: mean = 76.407 us, max = 232.777 us, min = 17.067 us, total = 50.505 ms HealthCheck - 658 total (0 active), Execution time: mean = 7.642 us, total = 5.028 ms, Queueing time: mean = 101.636 us, max = 399.856 us, min = 18.471 us, total = 66.876 ms GcsInMemoryStore.Get - 464 total (0 active), Execution time: mean = 22.358 us, total = 10.374 ms, Queueing time: mean = 13.674 us, max = 64.163 us, min = 4.068 us, total = 6.345 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 463 total (0 active), Execution time: mean = 53.482 us, total = 24.762 ms, Queueing time: mean = 107.727 us, max = 269.544 us, min = 13.883 us, total = 49.878 ms InternalKVGcsService.grpc_server.InternalKVGet - 463 total (0 active), Execution time: mean = 607.911 us, total = 281.463 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 395 total (0 active), Execution time: mean = 93.155 us, total = 36.796 ms, Queueing time: mean = 115.871 us, max = 296.469 us, min = 19.828 us, total = 45.769 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 395 total (0 active), Execution time: mean = 698.909 us, total = 276.069 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 198 total (1 active), Execution time: mean = 1.419 ms, total = 280.875 ms, Queueing time: mean = 72.240 us, max = 315.299 us, min = -0.000 s, total = 14.304 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms GCSServer.deadline_timer.debug_state_event_stats_print - 33 total (1 active, 1 running), Execution time: mean = 1.562 ms, total = 51.532 ms, Queueing time: mean = 57.814 us, max = 108.649 us, min = 15.007 us, total = 1.908 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:34:53,858 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 91351 total (1 active) Queueing time: mean = 51.049 us, max = 7.565 ms, min = 2.216 us, total = 4.663 s Execution time: mean = 259.229 us, total = 23.681 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 41517 total (0 active), Execution time: mean = 537.238 us, total = 22.305 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 41517 total (0 active), Execution time: mean = 27.603 us, total = 1.146 s, Queueing time: mean = 109.511 us, max = 7.565 ms, min = 3.372 us, total = 4.547 s event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 28.497 us, total = 225.668 ms, Queueing time: mean = 10.604 us, max = 4.983 ms, min = 2.216 us, total = 83.972 ms GcsTaskManager.GcJobSummary - 397 total (1 active), Execution time: mean = 11.663 us, total = 4.630 ms, Queueing time: mean = 82.523 us, max = 229.756 us, min = 21.532 us, total = 32.762 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:34:53,858 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 9843 total (25 active) Queueing time: mean = 20.147 us, max = 333.684 us, min = 2.330 us, total = 198.306 ms Execution time: mean = 4.368 s, total = 42998.453 s Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 31.284 us, total = 247.741 ms, Queueing time: mean = 11.508 us, max = 291.078 us, min = 2.330 us, total = 91.134 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 513 total (0 active), Execution time: mean = 43.294 us, total = 22.210 ms, Queueing time: mean = 109.012 us, max = 333.684 us, min = 14.442 us, total = 55.923 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 513 total (24 active), Execution time: mean = 83.816 s, total = 42997.837 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 420 total (0 active), Execution time: mean = 115.156 us, total = 48.366 ms, Queueing time: mean = 112.478 us, max = 247.252 us, min = 23.656 us, total = 47.241 ms InternalPubSubGcsService.grpc_server.GcsPublish - 420 total (0 active), Execution time: mean = 659.432 us, total = 276.962 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 604.476 us, total = 4.231 ms, Queueing time: mean = 84.400 us, max = 153.556 us, min = 52.516 us, total = 590.801 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:34:53,859 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 7945 total (0 active) Queueing time: mean = 10.512 us, max = 1.423 ms, min = 568.000 ns, total = 83.518 ms Execution time: mean = 27.651 us, total = 219.683 ms Event stats: event_loop_lag_probe - 7919 total (0 active), Execution time: mean = 27.546 us, total = 218.134 ms, Queueing time: mean = 10.309 us, max = 1.423 ms, min = 1.850 us, total = 81.639 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:35:53,859 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 407 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 432 - cumulative published bytes: 337564 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:35:53,860 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 24865 total (4 active) Queueing time: mean = 143.436 us, max = 279.303 ms, min = -0.000 s, total = 3.567 s Execution time: mean = 254.209 us, total = 6.321 s Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 24.164 us, total = 197.151 ms, Queueing time: mean = 10.862 us, max = 13.097 ms, min = 2.088 us, total = 88.627 ms GcsInMemoryStore.Put - 3308 total (0 active), Execution time: mean = 90.028 us, total = 297.813 ms, Queueing time: mean = 635.712 us, max = 278.545 ms, min = 2.310 us, total = 2.103 s RayletLoadPulled - 2040 total (1 active), Execution time: mean = 246.233 us, total = 502.315 ms, Queueing time: mean = 75.660 us, max = 460.894 us, min = 11.570 us, total = 154.347 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2037 total (0 active), Execution time: mean = 77.623 us, total = 158.119 ms, Queueing time: mean = 106.272 us, max = 330.146 us, min = 12.278 us, total = 216.476 ms NodeManagerService.grpc_client.GetResourceLoad - 2037 total (0 active), Execution time: mean = 1.512 ms, total = 3.080 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 828 total (0 active), Execution time: mean = 41.109 us, total = 34.038 ms, Queueing time: mean = 103.029 us, max = 228.583 us, min = 11.443 us, total = 85.308 ms InternalKVGcsService.grpc_server.InternalKVPut - 828 total (0 active), Execution time: mean = 572.048 us, total = 473.656 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 816 total (0 active), Execution time: mean = 34.059 us, total = 27.792 ms, Queueing time: mean = 118.951 us, max = 1.498 ms, min = 20.329 us, total = 97.064 ms NodeInfoGcsService.grpc_server.CheckAlive - 816 total (0 active), Execution time: mean = 596.961 us, total = 487.120 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 681 total (1 active), Execution time: mean = 9.683 us, total = 6.594 ms, Queueing time: mean = 76.394 us, max = 232.777 us, min = 17.067 us, total = 52.024 ms HealthCheck - 678 total (0 active), Execution time: mean = 7.619 us, total = 5.166 ms, Queueing time: mean = 101.597 us, max = 399.856 us, min = 18.471 us, total = 68.883 ms GcsInMemoryStore.Get - 476 total (0 active), Execution time: mean = 22.290 us, total = 10.610 ms, Queueing time: mean = 13.640 us, max = 64.163 us, min = 4.068 us, total = 6.492 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 475 total (0 active), Execution time: mean = 53.467 us, total = 25.397 ms, Queueing time: mean = 107.350 us, max = 269.544 us, min = 13.883 us, total = 50.991 ms InternalKVGcsService.grpc_server.InternalKVGet - 475 total (0 active), Execution time: mean = 607.045 us, total = 288.346 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 407 total (0 active), Execution time: mean = 93.287 us, total = 37.968 ms, Queueing time: mean = 115.990 us, max = 296.469 us, min = 19.828 us, total = 47.208 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 407 total (0 active), Execution time: mean = 700.106 us, total = 284.943 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 204 total (1 active), Execution time: mean = 1.423 ms, total = 290.268 ms, Queueing time: mean = 72.489 us, max = 315.299 us, min = -0.000 s, total = 14.788 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_event_stats_print - 34 total (1 active, 1 running), Execution time: mean = 1.563 ms, total = 53.146 ms, Queueing time: mean = 59.088 us, max = 108.649 us, min = 15.007 us, total = 2.009 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:35:53,860 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 94123 total (1 active) Queueing time: mean = 50.914 us, max = 7.565 ms, min = 2.216 us, total = 4.792 s Execution time: mean = 259.080 us, total = 24.385 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 42777 total (0 active), Execution time: mean = 536.925 us, total = 22.968 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 42777 total (0 active), Execution time: mean = 27.579 us, total = 1.180 s, Queueing time: mean = 109.214 us, max = 7.565 ms, min = 3.372 us, total = 4.672 s event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 28.536 us, total = 232.825 ms, Queueing time: mean = 10.610 us, max = 4.983 ms, min = 2.216 us, total = 86.569 ms GcsTaskManager.GcJobSummary - 409 total (1 active), Execution time: mean = 11.630 us, total = 4.757 ms, Queueing time: mean = 82.234 us, max = 229.756 us, min = 21.532 us, total = 33.634 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:35:53,860 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10131 total (25 active) Queueing time: mean = 20.097 us, max = 333.684 us, min = 2.330 us, total = 203.601 ms Execution time: mean = 4.250 s, total = 43058.629 s Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 31.274 us, total = 255.164 ms, Queueing time: mean = 11.507 us, max = 291.078 us, min = 2.330 us, total = 93.885 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 525 total (0 active), Execution time: mean = 43.307 us, total = 22.736 ms, Queueing time: mean = 108.683 us, max = 333.684 us, min = 14.442 us, total = 57.059 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 525 total (24 active), Execution time: mean = 82.015 s, total = 43057.995 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 432 total (0 active), Execution time: mean = 114.792 us, total = 49.590 ms, Queueing time: mean = 112.613 us, max = 247.252 us, min = 23.656 us, total = 48.649 ms InternalPubSubGcsService.grpc_server.GcsPublish - 432 total (0 active), Execution time: mean = 659.867 us, total = 285.063 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 7 total (1 active), Execution time: mean = 604.476 us, total = 4.231 ms, Queueing time: mean = 84.400 us, max = 153.556 us, min = 52.516 us, total = 590.801 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:35:53,861 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8185 total (0 active) Queueing time: mean = 10.496 us, max = 1.423 ms, min = 568.000 ns, total = 85.913 ms Execution time: mean = 27.630 us, total = 226.148 ms Event stats: event_loop_lag_probe - 8159 total (0 active), Execution time: mean = 27.528 us, total = 224.599 ms, Queueing time: mean = 10.299 us, max = 1.423 ms, min = 1.850 us, total = 84.033 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:36:53,861 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 419 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 444 - cumulative published bytes: 347392 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:36:53,862 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 25584 total (4 active) Queueing time: mean = 140.585 us, max = 279.303 ms, min = -0.000 s, total = 3.597 s Execution time: mean = 253.473 us, total = 6.485 s Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 24.142 us, total = 202.772 ms, Queueing time: mean = 10.811 us, max = 13.097 ms, min = 2.088 us, total = 90.804 ms GcsInMemoryStore.Put - 3404 total (0 active), Execution time: mean = 87.636 us, total = 298.314 ms, Queueing time: mean = 619.090 us, max = 278.545 ms, min = 2.310 us, total = 2.107 s RayletLoadPulled - 2100 total (1 active), Execution time: mean = 246.310 us, total = 517.251 ms, Queueing time: mean = 75.635 us, max = 460.894 us, min = 11.570 us, total = 158.833 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2097 total (0 active), Execution time: mean = 77.615 us, total = 162.758 ms, Queueing time: mean = 106.307 us, max = 330.146 us, min = 12.278 us, total = 222.926 ms NodeManagerService.grpc_client.GetResourceLoad - 2097 total (0 active), Execution time: mean = 1.509 ms, total = 3.165 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 852 total (0 active), Execution time: mean = 41.174 us, total = 35.080 ms, Queueing time: mean = 103.397 us, max = 228.583 us, min = 11.443 us, total = 88.094 ms InternalKVGcsService.grpc_server.InternalKVPut - 852 total (0 active), Execution time: mean = 570.483 us, total = 486.052 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 840 total (0 active), Execution time: mean = 34.015 us, total = 28.573 ms, Queueing time: mean = 118.887 us, max = 1.498 ms, min = 20.329 us, total = 99.865 ms NodeInfoGcsService.grpc_server.CheckAlive - 840 total (0 active), Execution time: mean = 595.112 us, total = 499.894 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 701 total (1 active), Execution time: mean = 9.666 us, total = 6.776 ms, Queueing time: mean = 76.269 us, max = 232.777 us, min = 17.067 us, total = 53.464 ms HealthCheck - 698 total (0 active), Execution time: mean = 7.620 us, total = 5.319 ms, Queueing time: mean = 101.874 us, max = 399.856 us, min = 18.471 us, total = 71.108 ms GcsInMemoryStore.Get - 488 total (0 active), Execution time: mean = 22.233 us, total = 10.850 ms, Queueing time: mean = 13.630 us, max = 64.163 us, min = 4.068 us, total = 6.651 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 487 total (0 active), Execution time: mean = 53.513 us, total = 26.061 ms, Queueing time: mean = 107.276 us, max = 269.544 us, min = 13.883 us, total = 52.244 ms InternalKVGcsService.grpc_server.InternalKVGet - 487 total (0 active), Execution time: mean = 604.997 us, total = 294.634 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 419 total (0 active), Execution time: mean = 93.213 us, total = 39.056 ms, Queueing time: mean = 116.113 us, max = 296.469 us, min = 19.828 us, total = 48.651 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 419 total (0 active), Execution time: mean = 697.626 us, total = 292.305 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 210 total (1 active), Execution time: mean = 1.424 ms, total = 299.072 ms, Queueing time: mean = 72.471 us, max = 315.299 us, min = -0.000 s, total = 15.219 ms GCSServer.deadline_timer.debug_state_event_stats_print - 35 total (1 active, 1 running), Execution time: mean = 1.572 ms, total = 55.037 ms, Queueing time: mean = 60.063 us, max = 108.649 us, min = 15.007 us, total = 2.102 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:36:53,862 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 96895 total (1 active) Queueing time: mean = 50.579 us, max = 7.565 ms, min = 2.216 us, total = 4.901 s Execution time: mean = 257.985 us, total = 24.997 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 44037 total (0 active), Execution time: mean = 534.586 us, total = 23.542 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 44037 total (0 active), Execution time: mean = 27.500 us, total = 1.211 s, Queueing time: mean = 108.482 us, max = 7.565 ms, min = 3.372 us, total = 4.777 s event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 28.572 us, total = 239.974 ms, Queueing time: mean = 10.599 us, max = 4.983 ms, min = 2.216 us, total = 89.023 ms GcsTaskManager.GcJobSummary - 421 total (1 active), Execution time: mean = 11.557 us, total = 4.866 ms, Queueing time: mean = 81.850 us, max = 229.756 us, min = 21.532 us, total = 34.459 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:36:53,862 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10422 total (25 active) Queueing time: mean = 19.981 us, max = 333.684 us, min = 2.330 us, total = 208.240 ms Execution time: mean = 4.195 s, total = 43718.798 s Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 31.254 us, total = 262.500 ms, Queueing time: mean = 11.492 us, max = 291.078 us, min = 2.330 us, total = 96.520 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 538 total (0 active), Execution time: mean = 43.320 us, total = 23.306 ms, Queueing time: mean = 107.765 us, max = 333.684 us, min = 14.442 us, total = 57.978 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 538 total (24 active), Execution time: mean = 81.260 s, total = 43718.149 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 444 total (0 active), Execution time: mean = 114.311 us, total = 50.754 ms, Queueing time: mean = 111.905 us, max = 247.252 us, min = 23.257 us, total = 49.686 ms InternalPubSubGcsService.grpc_server.GcsPublish - 444 total (0 active), Execution time: mean = 656.666 us, total = 291.560 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 534.634 us, total = 4.277 ms, Queueing time: mean = 79.895 us, max = 153.556 us, min = 48.361 us, total = 639.162 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:36:53,862 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8425 total (0 active) Queueing time: mean = 10.479 us, max = 1.423 ms, min = 568.000 ns, total = 88.287 ms Execution time: mean = 27.607 us, total = 232.591 ms Event stats: event_loop_lag_probe - 8399 total (0 active), Execution time: mean = 27.508 us, total = 231.042 ms, Queueing time: mean = 10.288 us, max = 1.423 ms, min = 1.850 us, total = 86.408 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:37:53,862 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 431 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 456 - cumulative published bytes: 357220 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:37:53,863 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 26303 total (4 active) Queueing time: mean = 137.784 us, max = 279.303 ms, min = -0.000 s, total = 3.624 s Execution time: mean = 252.115 us, total = 6.631 s Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 24.057 us, total = 207.826 ms, Queueing time: mean = 10.740 us, max = 13.097 ms, min = 2.088 us, total = 92.782 ms GcsInMemoryStore.Put - 3500 total (0 active), Execution time: mean = 85.373 us, total = 298.805 ms, Queueing time: mean = 603.311 us, max = 278.545 ms, min = 2.025 us, total = 2.112 s RayletLoadPulled - 2160 total (1 active), Execution time: mean = 245.975 us, total = 531.307 ms, Queueing time: mean = 75.459 us, max = 460.894 us, min = 11.570 us, total = 162.992 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2157 total (0 active), Execution time: mean = 77.457 us, total = 167.075 ms, Queueing time: mean = 105.918 us, max = 330.146 us, min = 11.910 us, total = 228.466 ms NodeManagerService.grpc_client.GetResourceLoad - 2157 total (0 active), Execution time: mean = 1.501 ms, total = 3.238 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 876 total (0 active), Execution time: mean = 41.025 us, total = 35.938 ms, Queueing time: mean = 103.416 us, max = 228.583 us, min = 11.443 us, total = 90.593 ms InternalKVGcsService.grpc_server.InternalKVPut - 876 total (0 active), Execution time: mean = 567.720 us, total = 497.323 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 864 total (0 active), Execution time: mean = 33.914 us, total = 29.302 ms, Queueing time: mean = 118.608 us, max = 1.498 ms, min = 20.329 us, total = 102.478 ms NodeInfoGcsService.grpc_server.CheckAlive - 864 total (0 active), Execution time: mean = 592.508 us, total = 511.927 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 721 total (1 active), Execution time: mean = 9.639 us, total = 6.950 ms, Queueing time: mean = 76.037 us, max = 232.777 us, min = 16.354 us, total = 54.823 ms HealthCheck - 718 total (0 active), Execution time: mean = 7.592 us, total = 5.451 ms, Queueing time: mean = 101.655 us, max = 399.856 us, min = 18.471 us, total = 72.988 ms GcsInMemoryStore.Get - 500 total (0 active), Execution time: mean = 22.118 us, total = 11.059 ms, Queueing time: mean = 13.587 us, max = 64.163 us, min = 3.701 us, total = 6.793 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 499 total (0 active), Execution time: mean = 53.395 us, total = 26.644 ms, Queueing time: mean = 107.124 us, max = 269.544 us, min = 13.461 us, total = 53.455 ms InternalKVGcsService.grpc_server.InternalKVGet - 499 total (0 active), Execution time: mean = 602.402 us, total = 300.599 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 431 total (0 active), Execution time: mean = 92.692 us, total = 39.950 ms, Queueing time: mean = 116.004 us, max = 296.469 us, min = 19.828 us, total = 49.998 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 431 total (0 active), Execution time: mean = 693.473 us, total = 298.887 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 216 total (1 active), Execution time: mean = 1.425 ms, total = 307.848 ms, Queueing time: mean = 72.418 us, max = 315.299 us, min = -0.000 s, total = 15.642 ms GCSServer.deadline_timer.debug_state_event_stats_print - 36 total (1 active, 1 running), Execution time: mean = 1.576 ms, total = 56.726 ms, Queueing time: mean = 60.237 us, max = 108.649 us, min = 15.007 us, total = 2.169 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:37:53,864 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 99667 total (1 active) Queueing time: mean = 50.083 us, max = 7.565 ms, min = 2.216 us, total = 4.992 s Execution time: mean = 256.037 us, total = 25.518 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 45297 total (0 active), Execution time: mean = 530.518 us, total = 24.031 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 45297 total (0 active), Execution time: mean = 27.291 us, total = 1.236 s, Queueing time: mean = 107.402 us, max = 7.565 ms, min = 3.372 us, total = 4.865 s event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 28.514 us, total = 246.332 ms, Queueing time: mean = 10.562 us, max = 4.983 ms, min = 2.216 us, total = 91.249 ms GcsTaskManager.GcJobSummary - 433 total (1 active), Execution time: mean = 11.462 us, total = 4.963 ms, Queueing time: mean = 81.596 us, max = 229.756 us, min = 21.532 us, total = 35.331 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:37:53,864 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10710 total (25 active) Queueing time: mean = 19.817 us, max = 333.684 us, min = 2.330 us, total = 212.243 ms Execution time: mean = 4.088 s, total = 43778.929 s Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 31.137 us, total = 268.988 ms, Queueing time: mean = 11.441 us, max = 291.078 us, min = 2.330 us, total = 98.841 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 550 total (0 active), Execution time: mean = 43.228 us, total = 23.776 ms, Queueing time: mean = 106.862 us, max = 333.684 us, min = 14.442 us, total = 58.774 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 550 total (24 active), Execution time: mean = 79.597 s, total = 43778.266 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 456 total (0 active), Execution time: mean = 114.384 us, total = 52.159 ms, Queueing time: mean = 110.902 us, max = 247.252 us, min = 23.257 us, total = 50.571 ms InternalPubSubGcsService.grpc_server.GcsPublish - 456 total (0 active), Execution time: mean = 652.486 us, total = 297.534 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 534.634 us, total = 4.277 ms, Queueing time: mean = 79.895 us, max = 153.556 us, min = 48.361 us, total = 639.162 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:37:53,864 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8665 total (0 active) Queueing time: mean = 10.410 us, max = 1.423 ms, min = 568.000 ns, total = 90.203 ms Execution time: mean = 27.444 us, total = 237.804 ms Event stats: event_loop_lag_probe - 8639 total (0 active), Execution time: mean = 27.347 us, total = 236.255 ms, Queueing time: mean = 10.224 us, max = 1.423 ms, min = 1.850 us, total = 88.323 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:38:53,864 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 443 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 468 - cumulative published bytes: 367048 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:38:53,865 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 27022 total (4 active) Queueing time: mean = 135.111 us, max = 279.303 ms, min = -0.000 s, total = 3.651 s Execution time: mean = 250.512 us, total = 6.769 s Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 23.932 us, total = 212.488 ms, Queueing time: mean = 10.656 us, max = 13.097 ms, min = 2.088 us, total = 94.619 ms GcsInMemoryStore.Put - 3596 total (0 active), Execution time: mean = 83.236 us, total = 299.315 ms, Queueing time: mean = 588.301 us, max = 278.545 ms, min = 2.025 us, total = 2.116 s RayletLoadPulled - 2220 total (1 active), Execution time: mean = 245.037 us, total = 543.983 ms, Queueing time: mean = 75.096 us, max = 460.894 us, min = 11.570 us, total = 166.714 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2217 total (0 active), Execution time: mean = 77.336 us, total = 171.454 ms, Queueing time: mean = 105.773 us, max = 330.146 us, min = 11.910 us, total = 234.498 ms NodeManagerService.grpc_client.GetResourceLoad - 2217 total (0 active), Execution time: mean = 1.491 ms, total = 3.306 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 900 total (0 active), Execution time: mean = 41.594 us, total = 37.435 ms, Queueing time: mean = 103.294 us, max = 228.583 us, min = 11.443 us, total = 92.965 ms InternalKVGcsService.grpc_server.InternalKVPut - 900 total (0 active), Execution time: mean = 564.061 us, total = 507.655 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 888 total (0 active), Execution time: mean = 33.802 us, total = 30.016 ms, Queueing time: mean = 118.529 us, max = 1.498 ms, min = 20.329 us, total = 105.253 ms NodeInfoGcsService.grpc_server.CheckAlive - 888 total (0 active), Execution time: mean = 589.731 us, total = 523.681 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 741 total (1 active), Execution time: mean = 9.608 us, total = 7.120 ms, Queueing time: mean = 75.586 us, max = 232.777 us, min = 16.354 us, total = 56.009 ms HealthCheck - 738 total (0 active), Execution time: mean = 7.575 us, total = 5.590 ms, Queueing time: mean = 101.454 us, max = 399.856 us, min = 16.806 us, total = 74.873 ms GcsInMemoryStore.Get - 512 total (0 active), Execution time: mean = 22.019 us, total = 11.274 ms, Queueing time: mean = 13.554 us, max = 64.163 us, min = 3.701 us, total = 6.939 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 511 total (0 active), Execution time: mean = 53.368 us, total = 27.271 ms, Queueing time: mean = 106.960 us, max = 269.544 us, min = 13.461 us, total = 54.657 ms InternalKVGcsService.grpc_server.InternalKVGet - 511 total (0 active), Execution time: mean = 598.664 us, total = 305.917 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 443 total (0 active), Execution time: mean = 92.567 us, total = 41.007 ms, Queueing time: mean = 115.641 us, max = 296.469 us, min = 19.828 us, total = 51.229 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 443 total (0 active), Execution time: mean = 687.580 us, total = 304.598 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 222 total (1 active), Execution time: mean = 1.424 ms, total = 316.191 ms, Queueing time: mean = 72.318 us, max = 315.299 us, min = -0.000 s, total = 16.055 ms GCSServer.deadline_timer.debug_state_event_stats_print - 37 total (1 active, 1 running), Execution time: mean = 1.575 ms, total = 58.273 ms, Queueing time: mean = 60.776 us, max = 108.649 us, min = 15.007 us, total = 2.249 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:38:53,865 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 102439 total (1 active) Queueing time: mean = 49.437 us, max = 7.565 ms, min = 2.216 us, total = 5.064 s Execution time: mean = 253.582 us, total = 25.977 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 46557 total (0 active), Execution time: mean = 525.360 us, total = 24.459 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 46557 total (0 active), Execution time: mean = 27.057 us, total = 1.260 s, Queueing time: mean = 105.989 us, max = 7.565 ms, min = 3.372 us, total = 4.935 s event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 28.468 us, total = 252.764 ms, Queueing time: mean = 10.531 us, max = 4.983 ms, min = 2.216 us, total = 93.508 ms GcsTaskManager.GcJobSummary - 445 total (1 active), Execution time: mean = 11.416 us, total = 5.080 ms, Queueing time: mean = 81.203 us, max = 229.756 us, min = 21.532 us, total = 36.135 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:38:53,865 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 10998 total (25 active) Queueing time: mean = 19.635 us, max = 333.684 us, min = 2.330 us, total = 215.942 ms Execution time: mean = 3.986 s, total = 43839.154 s Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 30.996 us, total = 275.218 ms, Queueing time: mean = 11.383 us, max = 291.078 us, min = 2.330 us, total = 101.070 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 562 total (0 active), Execution time: mean = 43.219 us, total = 24.289 ms, Queueing time: mean = 105.659 us, max = 333.684 us, min = 10.245 us, total = 59.381 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 562 total (24 active), Execution time: mean = 78.004 s, total = 43838.476 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 468 total (0 active), Execution time: mean = 113.848 us, total = 53.281 ms, Queueing time: mean = 109.905 us, max = 247.252 us, min = 21.083 us, total = 51.435 ms InternalPubSubGcsService.grpc_server.GcsPublish - 468 total (0 active), Execution time: mean = 647.817 us, total = 303.178 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 534.634 us, total = 4.277 ms, Queueing time: mean = 79.895 us, max = 153.556 us, min = 48.361 us, total = 639.162 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:38:53,866 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 8905 total (0 active) Queueing time: mean = 10.341 us, max = 1.423 ms, min = 568.000 ns, total = 92.086 ms Execution time: mean = 27.289 us, total = 243.013 ms Event stats: event_loop_lag_probe - 8879 total (0 active), Execution time: mean = 27.195 us, total = 241.464 ms, Queueing time: mean = 10.160 us, max = 1.423 ms, min = 1.850 us, total = 90.207 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:39:53,866 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 455 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 480 - cumulative published bytes: 376876 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:39:53,867 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 27741 total (4 active) Queueing time: mean = 132.577 us, max = 279.303 ms, min = -0.000 s, total = 3.678 s Execution time: mean = 249.297 us, total = 6.916 s Event stats: event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 23.825 us, total = 217.263 ms, Queueing time: mean = 10.587 us, max = 13.097 ms, min = 2.088 us, total = 96.539 ms GcsInMemoryStore.Put - 3692 total (0 active), Execution time: mean = 81.206 us, total = 299.813 ms, Queueing time: mean = 574.012 us, max = 278.545 ms, min = 2.025 us, total = 2.119 s RayletLoadPulled - 2280 total (1 active), Execution time: mean = 244.404 us, total = 557.240 ms, Queueing time: mean = 74.836 us, max = 460.894 us, min = 11.570 us, total = 170.626 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2277 total (0 active), Execution time: mean = 77.065 us, total = 175.476 ms, Queueing time: mean = 105.487 us, max = 330.146 us, min = 11.910 us, total = 240.194 ms NodeManagerService.grpc_client.GetResourceLoad - 2277 total (0 active), Execution time: mean = 1.484 ms, total = 3.380 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 924 total (0 active), Execution time: mean = 41.602 us, total = 38.440 ms, Queueing time: mean = 103.423 us, max = 228.583 us, min = 11.443 us, total = 95.563 ms InternalKVGcsService.grpc_server.InternalKVPut - 924 total (0 active), Execution time: mean = 562.412 us, total = 519.668 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 912 total (0 active), Execution time: mean = 33.644 us, total = 30.683 ms, Queueing time: mean = 118.413 us, max = 1.498 ms, min = 20.329 us, total = 107.993 ms NodeInfoGcsService.grpc_server.CheckAlive - 912 total (0 active), Execution time: mean = 587.221 us, total = 535.545 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 761 total (1 active), Execution time: mean = 9.570 us, total = 7.283 ms, Queueing time: mean = 75.265 us, max = 232.777 us, min = 16.354 us, total = 57.277 ms HealthCheck - 758 total (0 active), Execution time: mean = 7.549 us, total = 5.722 ms, Queueing time: mean = 101.265 us, max = 399.856 us, min = 16.806 us, total = 76.759 ms GcsInMemoryStore.Get - 524 total (0 active), Execution time: mean = 21.916 us, total = 11.484 ms, Queueing time: mean = 13.543 us, max = 64.163 us, min = 3.701 us, total = 7.097 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 523 total (0 active), Execution time: mean = 53.396 us, total = 27.926 ms, Queueing time: mean = 106.697 us, max = 269.544 us, min = 13.461 us, total = 55.803 ms InternalKVGcsService.grpc_server.InternalKVGet - 523 total (0 active), Execution time: mean = 595.878 us, total = 311.644 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 455 total (0 active), Execution time: mean = 92.274 us, total = 41.985 ms, Queueing time: mean = 115.560 us, max = 296.469 us, min = 19.828 us, total = 52.580 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 455 total (0 active), Execution time: mean = 684.690 us, total = 311.534 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 228 total (1 active), Execution time: mean = 1.421 ms, total = 324.020 ms, Queueing time: mean = 72.130 us, max = 315.299 us, min = -0.000 s, total = 16.446 ms GCSServer.deadline_timer.debug_state_event_stats_print - 38 total (1 active, 1 running), Execution time: mean = 1.576 ms, total = 59.901 ms, Queueing time: mean = 60.739 us, max = 108.649 us, min = 15.007 us, total = 2.308 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:39:53,867 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 105210 total (1 active) Queueing time: mean = 49.053 us, max = 7.565 ms, min = 2.216 us, total = 5.161 s Execution time: mean = 252.140 us, total = 26.528 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 47817 total (0 active), Execution time: mean = 522.343 us, total = 24.977 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 47817 total (0 active), Execution time: mean = 26.913 us, total = 1.287 s, Queueing time: mean = 105.154 us, max = 7.565 ms, min = 3.372 us, total = 5.028 s event_loop_lag_probe - 9118 total (0 active), Execution time: mean = 28.367 us, total = 258.653 ms, Queueing time: mean = 10.483 us, max = 4.983 ms, min = 2.216 us, total = 95.587 ms GcsTaskManager.GcJobSummary - 457 total (1 active), Execution time: mean = 11.326 us, total = 5.176 ms, Queueing time: mean = 81.080 us, max = 229.756 us, min = 21.532 us, total = 37.053 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:39:53,867 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11285 total (25 active) Queueing time: mean = 19.509 us, max = 333.684 us, min = 2.330 us, total = 220.161 ms Execution time: mean = 3.890 s, total = 43899.272 s Event stats: event_loop_lag_probe - 9118 total (0 active), Execution time: mean = 30.900 us, total = 281.749 ms, Queueing time: mean = 11.345 us, max = 291.078 us, min = 2.330 us, total = 103.442 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 574 total (0 active), Execution time: mean = 43.101 us, total = 24.740 ms, Queueing time: mean = 105.003 us, max = 333.684 us, min = 10.245 us, total = 60.272 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 574 total (24 active), Execution time: mean = 76.478 s, total = 43898.581 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 480 total (0 active), Execution time: mean = 113.559 us, total = 54.508 ms, Queueing time: mean = 109.147 us, max = 247.252 us, min = 21.083 us, total = 52.391 ms InternalPubSubGcsService.grpc_server.GcsPublish - 480 total (0 active), Execution time: mean = 644.408 us, total = 309.316 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 534.634 us, total = 4.277 ms, Queueing time: mean = 79.895 us, max = 153.556 us, min = 48.361 us, total = 639.162 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:39:53,867 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9145 total (0 active) Queueing time: mean = 10.276 us, max = 1.423 ms, min = 568.000 ns, total = 93.971 ms Execution time: mean = 27.148 us, total = 248.265 ms Event stats: event_loop_lag_probe - 9119 total (0 active), Execution time: mean = 27.055 us, total = 246.716 ms, Queueing time: mean = 10.099 us, max = 1.423 ms, min = 1.850 us, total = 92.092 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:40:53,868 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 467 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 492 - cumulative published bytes: 386704 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:40:53,868 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 28460 total (4 active) Queueing time: mean = 130.221 us, max = 279.303 ms, min = -0.000 s, total = 3.706 s Execution time: mean = 247.738 us, total = 7.051 s Event stats: event_loop_lag_probe - 9359 total (0 active), Execution time: mean = 23.754 us, total = 222.309 ms, Queueing time: mean = 10.522 us, max = 13.097 ms, min = 2.088 us, total = 98.478 ms GcsInMemoryStore.Put - 3788 total (0 active), Execution time: mean = 79.272 us, total = 300.281 ms, Queueing time: mean = 560.484 us, max = 278.545 ms, min = 2.025 us, total = 2.123 s RayletLoadPulled - 2340 total (1 active), Execution time: mean = 243.521 us, total = 569.840 ms, Queueing time: mean = 74.564 us, max = 460.894 us, min = 11.570 us, total = 174.479 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2337 total (0 active), Execution time: mean = 76.787 us, total = 179.451 ms, Queueing time: mean = 105.483 us, max = 330.146 us, min = 11.910 us, total = 246.513 ms NodeManagerService.grpc_client.GetResourceLoad - 2337 total (0 active), Execution time: mean = 1.475 ms, total = 3.447 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 948 total (0 active), Execution time: mean = 41.645 us, total = 39.479 ms, Queueing time: mean = 103.548 us, max = 228.583 us, min = 11.443 us, total = 98.164 ms InternalKVGcsService.grpc_server.InternalKVPut - 948 total (0 active), Execution time: mean = 558.445 us, total = 529.406 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 936 total (0 active), Execution time: mean = 33.495 us, total = 31.351 ms, Queueing time: mean = 118.358 us, max = 1.498 ms, min = 20.329 us, total = 110.783 ms NodeInfoGcsService.grpc_server.CheckAlive - 936 total (0 active), Execution time: mean = 583.473 us, total = 546.131 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 781 total (1 active), Execution time: mean = 9.513 us, total = 7.430 ms, Queueing time: mean = 75.232 us, max = 232.777 us, min = 16.354 us, total = 58.756 ms HealthCheck - 778 total (0 active), Execution time: mean = 7.523 us, total = 5.853 ms, Queueing time: mean = 101.498 us, max = 399.856 us, min = 16.806 us, total = 78.965 ms GcsInMemoryStore.Get - 536 total (0 active), Execution time: mean = 21.805 us, total = 11.687 ms, Queueing time: mean = 13.514 us, max = 64.163 us, min = 3.701 us, total = 7.244 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 535 total (0 active), Execution time: mean = 53.398 us, total = 28.568 ms, Queueing time: mean = 106.656 us, max = 269.544 us, min = 13.461 us, total = 57.061 ms InternalKVGcsService.grpc_server.InternalKVGet - 535 total (0 active), Execution time: mean = 592.396 us, total = 316.932 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 467 total (0 active), Execution time: mean = 91.996 us, total = 42.962 ms, Queueing time: mean = 115.537 us, max = 296.469 us, min = 19.828 us, total = 53.956 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 467 total (0 active), Execution time: mean = 680.084 us, total = 317.599 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 234 total (1 active), Execution time: mean = 1.420 ms, total = 332.286 ms, Queueing time: mean = 71.873 us, max = 315.299 us, min = -0.000 s, total = 16.818 ms GCSServer.deadline_timer.debug_state_event_stats_print - 39 total (1 active, 1 running), Execution time: mean = 1.579 ms, total = 61.591 ms, Queueing time: mean = 60.882 us, max = 108.649 us, min = 15.007 us, total = 2.374 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:40:53,868 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 107982 total (1 active) Queueing time: mean = 48.306 us, max = 7.565 ms, min = 2.216 us, total = 5.216 s Execution time: mean = 249.290 us, total = 26.919 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 49077 total (0 active), Execution time: mean = 516.408 us, total = 25.344 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 49077 total (0 active), Execution time: mean = 26.600 us, total = 1.305 s, Queueing time: mean = 103.530 us, max = 7.565 ms, min = 3.372 us, total = 5.081 s event_loop_lag_probe - 9358 total (0 active), Execution time: mean = 28.251 us, total = 264.373 ms, Queueing time: mean = 10.421 us, max = 4.983 ms, min = 2.216 us, total = 97.523 ms GcsTaskManager.GcJobSummary - 469 total (1 active), Execution time: mean = 11.207 us, total = 5.256 ms, Queueing time: mean = 80.116 us, max = 229.756 us, min = 18.742 us, total = 37.575 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:40:53,868 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11573 total (25 active) Queueing time: mean = 19.308 us, max = 333.684 us, min = 2.330 us, total = 223.452 ms Execution time: mean = 3.798 s, total = 43959.473 s Event stats: event_loop_lag_probe - 9358 total (0 active), Execution time: mean = 30.753 us, total = 287.790 ms, Queueing time: mean = 11.288 us, max = 291.078 us, min = 2.330 us, total = 105.637 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 586 total (0 active), Execution time: mean = 43.021 us, total = 25.210 ms, Queueing time: mean = 103.674 us, max = 333.684 us, min = 10.245 us, total = 60.753 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 586 total (24 active), Execution time: mean = 75.015 s, total = 43958.770 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 492 total (0 active), Execution time: mean = 113.016 us, total = 55.604 ms, Queueing time: mean = 107.734 us, max = 247.252 us, min = 21.083 us, total = 53.005 ms InternalPubSubGcsService.grpc_server.GcsPublish - 492 total (0 active), Execution time: mean = 638.405 us, total = 314.095 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 8 total (1 active), Execution time: mean = 534.634 us, total = 4.277 ms, Queueing time: mean = 79.895 us, max = 153.556 us, min = 48.361 us, total = 639.162 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:40:53,868 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9385 total (0 active) Queueing time: mean = 10.226 us, max = 1.423 ms, min = 568.000 ns, total = 95.972 ms Execution time: mean = 27.033 us, total = 253.707 ms Event stats: event_loop_lag_probe - 9359 total (0 active), Execution time: mean = 26.943 us, total = 252.158 ms, Queueing time: mean = 10.054 us, max = 1.423 ms, min = 1.850 us, total = 94.093 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:41:53,869 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 479 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 505 - cumulative published bytes: 396566 - current buffered bytes: 153 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:41:53,869 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 29179 total (4 active) Queueing time: mean = 128.042 us, max = 279.303 ms, min = -0.000 s, total = 3.736 s Execution time: mean = 247.088 us, total = 7.210 s Event stats: event_loop_lag_probe - 9599 total (0 active), Execution time: mean = 23.765 us, total = 228.122 ms, Queueing time: mean = 10.497 us, max = 13.097 ms, min = 2.088 us, total = 100.757 ms GcsInMemoryStore.Put - 3884 total (0 active), Execution time: mean = 77.430 us, total = 300.739 ms, Queueing time: mean = 547.704 us, max = 278.545 ms, min = 2.025 us, total = 2.127 s RayletLoadPulled - 2400 total (1 active), Execution time: mean = 243.673 us, total = 584.815 ms, Queueing time: mean = 74.521 us, max = 460.894 us, min = 11.570 us, total = 178.850 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2397 total (0 active), Execution time: mean = 76.758 us, total = 183.988 ms, Queueing time: mean = 105.682 us, max = 330.146 us, min = 11.910 us, total = 253.319 ms NodeManagerService.grpc_client.GetResourceLoad - 2397 total (0 active), Execution time: mean = 1.472 ms, total = 3.529 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 972 total (0 active), Execution time: mean = 41.563 us, total = 40.399 ms, Queueing time: mean = 103.492 us, max = 228.583 us, min = 11.443 us, total = 100.594 ms InternalKVGcsService.grpc_server.InternalKVPut - 972 total (0 active), Execution time: mean = 556.551 us, total = 540.968 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 960 total (0 active), Execution time: mean = 33.453 us, total = 32.114 ms, Queueing time: mean = 118.268 us, max = 1.498 ms, min = 20.329 us, total = 113.537 ms NodeInfoGcsService.grpc_server.CheckAlive - 960 total (0 active), Execution time: mean = 581.951 us, total = 558.673 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 801 total (1 active), Execution time: mean = 9.483 us, total = 7.595 ms, Queueing time: mean = 75.281 us, max = 232.777 us, min = 16.354 us, total = 60.300 ms HealthCheck - 798 total (0 active), Execution time: mean = 7.518 us, total = 5.999 ms, Queueing time: mean = 101.696 us, max = 399.856 us, min = 16.806 us, total = 81.154 ms GcsInMemoryStore.Get - 548 total (0 active), Execution time: mean = 21.720 us, total = 11.902 ms, Queueing time: mean = 13.503 us, max = 64.163 us, min = 3.701 us, total = 7.400 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 547 total (0 active), Execution time: mean = 53.393 us, total = 29.206 ms, Queueing time: mean = 106.661 us, max = 269.544 us, min = 13.461 us, total = 58.343 ms InternalKVGcsService.grpc_server.InternalKVGet - 547 total (0 active), Execution time: mean = 591.007 us, total = 323.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 479 total (0 active), Execution time: mean = 91.902 us, total = 44.021 ms, Queueing time: mean = 115.861 us, max = 296.469 us, min = 19.828 us, total = 55.498 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 479 total (0 active), Execution time: mean = 678.838 us, total = 325.164 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 240 total (1 active), Execution time: mean = 1.421 ms, total = 341.137 ms, Queueing time: mean = 72.052 us, max = 315.299 us, min = -0.000 s, total = 17.292 ms GCSServer.deadline_timer.debug_state_event_stats_print - 40 total (1 active, 1 running), Execution time: mean = 1.564 ms, total = 62.576 ms, Queueing time: mean = 60.589 us, max = 108.649 us, min = 15.007 us, total = 2.424 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:41:53,870 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 110754 total (1 active) Queueing time: mean = 48.045 us, max = 7.565 ms, min = 2.216 us, total = 5.321 s Execution time: mean = 248.265 us, total = 27.496 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 50337 total (0 active), Execution time: mean = 514.241 us, total = 25.885 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 50337 total (0 active), Execution time: mean = 26.496 us, total = 1.334 s, Queueing time: mean = 102.956 us, max = 7.565 ms, min = 3.372 us, total = 5.183 s event_loop_lag_probe - 9598 total (0 active), Execution time: mean = 28.325 us, total = 271.864 ms, Queueing time: mean = 10.429 us, max = 4.983 ms, min = 2.216 us, total = 100.099 ms GcsTaskManager.GcJobSummary - 481 total (1 active), Execution time: mean = 11.148 us, total = 5.362 ms, Queueing time: mean = 80.029 us, max = 229.756 us, min = 18.742 us, total = 38.494 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:41:53,870 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 11910 total (25 active) Queueing time: mean = 19.295 us, max = 333.684 us, min = 2.330 us, total = 229.800 ms Execution time: mean = 4.804 s, total = 57219.540 s Event stats: event_loop_lag_probe - 9598 total (0 active), Execution time: mean = 30.688 us, total = 294.543 ms, Queueing time: mean = 11.272 us, max = 291.078 us, min = 2.330 us, total = 108.190 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 621 total (0 active), Execution time: mean = 42.375 us, total = 26.315 ms, Queueing time: mean = 102.092 us, max = 333.684 us, min = 8.944 us, total = 63.399 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 621 total (24 active), Execution time: mean = 92.140 s, total = 57218.819 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 505 total (0 active), Execution time: mean = 112.718 us, total = 56.923 ms, Queueing time: mean = 107.067 us, max = 247.252 us, min = 21.083 us, total = 54.069 ms InternalPubSubGcsService.grpc_server.GcsPublish - 505 total (0 active), Execution time: mean = 634.947 us, total = 320.648 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 680.274 us, total = 6.122 ms, Queueing time: mean = 80.517 us, max = 153.556 us, min = 48.361 us, total = 724.654 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:41:53,870 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9625 total (0 active) Queueing time: mean = 10.215 us, max = 1.423 ms, min = 568.000 ns, total = 98.317 ms Execution time: mean = 27.024 us, total = 260.106 ms Event stats: event_loop_lag_probe - 9599 total (0 active), Execution time: mean = 26.936 us, total = 258.557 ms, Queueing time: mean = 10.047 us, max = 1.423 ms, min = 1.850 us, total = 96.437 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:42:53,870 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 491 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 517 - cumulative published bytes: 406394 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:42:53,871 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 29897 total (4 active) Queueing time: mean = 126.045 us, max = 279.303 ms, min = -0.000 s, total = 3.768 s Execution time: mean = 247.267 us, total = 7.393 s Event stats: event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 23.836 us, total = 234.503 ms, Queueing time: mean = 10.486 us, max = 13.097 ms, min = 2.088 us, total = 103.161 ms GcsInMemoryStore.Put - 3980 total (0 active), Execution time: mean = 75.700 us, total = 301.286 ms, Queueing time: mean = 535.659 us, max = 278.545 ms, min = 2.025 us, total = 2.132 s RayletLoadPulled - 2460 total (1 active), Execution time: mean = 244.189 us, total = 600.705 ms, Queueing time: mean = 74.696 us, max = 460.894 us, min = 11.570 us, total = 183.752 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2457 total (0 active), Execution time: mean = 76.916 us, total = 188.983 ms, Queueing time: mean = 105.759 us, max = 330.146 us, min = 11.910 us, total = 259.850 ms NodeManagerService.grpc_client.GetResourceLoad - 2457 total (0 active), Execution time: mean = 1.474 ms, total = 3.621 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 996 total (0 active), Execution time: mean = 41.707 us, total = 41.540 ms, Queueing time: mean = 103.965 us, max = 228.583 us, min = 11.443 us, total = 103.549 ms InternalKVGcsService.grpc_server.InternalKVPut - 996 total (0 active), Execution time: mean = 558.519 us, total = 556.285 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 984 total (0 active), Execution time: mean = 33.487 us, total = 32.952 ms, Queueing time: mean = 118.623 us, max = 1.498 ms, min = 20.329 us, total = 116.725 ms NodeInfoGcsService.grpc_server.CheckAlive - 984 total (0 active), Execution time: mean = 582.943 us, total = 573.616 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 821 total (1 active), Execution time: mean = 9.501 us, total = 7.800 ms, Queueing time: mean = 75.541 us, max = 232.777 us, min = 16.354 us, total = 62.019 ms HealthCheck - 818 total (0 active), Execution time: mean = 7.522 us, total = 6.153 ms, Queueing time: mean = 101.904 us, max = 399.856 us, min = 16.806 us, total = 83.357 ms GcsInMemoryStore.Get - 560 total (0 active), Execution time: mean = 21.702 us, total = 12.153 ms, Queueing time: mean = 13.520 us, max = 64.163 us, min = 3.701 us, total = 7.571 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 559 total (0 active), Execution time: mean = 53.674 us, total = 30.004 ms, Queueing time: mean = 106.811 us, max = 269.544 us, min = 13.461 us, total = 59.707 ms InternalKVGcsService.grpc_server.InternalKVGet - 559 total (0 active), Execution time: mean = 592.288 us, total = 331.089 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 491 total (0 active), Execution time: mean = 91.949 us, total = 45.147 ms, Queueing time: mean = 116.382 us, max = 296.469 us, min = 19.828 us, total = 57.144 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 491 total (0 active), Execution time: mean = 680.947 us, total = 334.345 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 246 total (1 active), Execution time: mean = 1.425 ms, total = 350.547 ms, Queueing time: mean = 72.129 us, max = 315.299 us, min = -0.000 s, total = 17.744 ms GCSServer.deadline_timer.debug_state_event_stats_print - 41 total (1 active, 1 running), Execution time: mean = 1.562 ms, total = 64.060 ms, Queueing time: mean = 60.711 us, max = 108.649 us, min = 15.007 us, total = 2.489 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:42:53,871 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 113526 total (1 active) Queueing time: mean = 48.100 us, max = 7.565 ms, min = 2.216 us, total = 5.461 s Execution time: mean = 248.652 us, total = 28.229 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 51597 total (0 active), Execution time: mean = 515.038 us, total = 26.574 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 51597 total (0 active), Execution time: mean = 26.526 us, total = 1.369 s, Queueing time: mean = 103.068 us, max = 7.565 ms, min = 3.372 us, total = 5.318 s event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 28.451 us, total = 279.897 ms, Queueing time: mean = 10.466 us, max = 4.983 ms, min = 2.216 us, total = 102.966 ms GcsTaskManager.GcJobSummary - 493 total (1 active), Execution time: mean = 11.217 us, total = 5.530 ms, Queueing time: mean = 80.190 us, max = 229.756 us, min = 18.742 us, total = 39.534 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:42:53,871 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12198 total (25 active) Queueing time: mean = 19.297 us, max = 333.684 us, min = 2.330 us, total = 235.385 ms Execution time: mean = 4.696 s, total = 57279.760 s Event stats: event_loop_lag_probe - 9838 total (0 active), Execution time: mean = 30.793 us, total = 302.941 ms, Queueing time: mean = 11.302 us, max = 291.078 us, min = 2.330 us, total = 111.190 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 633 total (0 active), Execution time: mean = 42.565 us, total = 26.943 ms, Queueing time: mean = 101.938 us, max = 333.684 us, min = 8.944 us, total = 64.527 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 633 total (24 active), Execution time: mean = 90.488 s, total = 57279.020 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 517 total (0 active), Execution time: mean = 112.938 us, total = 58.389 ms, Queueing time: mean = 107.401 us, max = 247.252 us, min = 21.083 us, total = 55.526 ms InternalPubSubGcsService.grpc_server.GcsPublish - 517 total (0 active), Execution time: mean = 635.970 us, total = 328.797 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 680.274 us, total = 6.122 ms, Queueing time: mean = 80.517 us, max = 153.556 us, min = 48.361 us, total = 724.654 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:42:53,871 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 9865 total (0 active) Queueing time: mean = 10.227 us, max = 1.423 ms, min = 568.000 ns, total = 100.894 ms Execution time: mean = 27.086 us, total = 267.207 ms Event stats: event_loop_lag_probe - 9839 total (0 active), Execution time: mean = 27.001 us, total = 265.658 ms, Queueing time: mean = 10.063 us, max = 1.423 ms, min = 1.850 us, total = 99.015 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:43:53,872 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 503 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 529 - cumulative published bytes: 416222 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:43:53,873 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 30616 total (4 active) Queueing time: mean = 124.047 us, max = 279.303 ms, min = -0.000 s, total = 3.798 s Execution time: mean = 247.221 us, total = 7.569 s Event stats: event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 23.849 us, total = 240.348 ms, Queueing time: mean = 10.454 us, max = 13.097 ms, min = 2.088 us, total = 105.357 ms GcsInMemoryStore.Put - 4076 total (0 active), Execution time: mean = 74.041 us, total = 301.792 ms, Queueing time: mean = 524.139 us, max = 278.545 ms, min = 2.025 us, total = 2.136 s RayletLoadPulled - 2520 total (1 active), Execution time: mean = 244.555 us, total = 616.279 ms, Queueing time: mean = 74.904 us, max = 472.090 us, min = 11.570 us, total = 188.758 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2517 total (0 active), Execution time: mean = 76.884 us, total = 193.517 ms, Queueing time: mean = 105.693 us, max = 330.146 us, min = 11.910 us, total = 266.030 ms NodeManagerService.grpc_client.GetResourceLoad - 2517 total (0 active), Execution time: mean = 1.475 ms, total = 3.714 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1020 total (0 active), Execution time: mean = 41.847 us, total = 42.684 ms, Queueing time: mean = 103.865 us, max = 228.583 us, min = 11.443 us, total = 105.942 ms InternalKVGcsService.grpc_server.InternalKVPut - 1020 total (0 active), Execution time: mean = 558.412 us, total = 569.580 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1008 total (0 active), Execution time: mean = 33.528 us, total = 33.796 ms, Queueing time: mean = 118.380 us, max = 1.498 ms, min = 20.329 us, total = 119.327 ms NodeInfoGcsService.grpc_server.CheckAlive - 1008 total (0 active), Execution time: mean = 583.357 us, total = 588.024 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 841 total (1 active), Execution time: mean = 9.480 us, total = 7.973 ms, Queueing time: mean = 75.581 us, max = 260.630 us, min = 16.354 us, total = 63.564 ms HealthCheck - 838 total (0 active), Execution time: mean = 7.512 us, total = 6.295 ms, Queueing time: mean = 101.729 us, max = 399.856 us, min = 16.806 us, total = 85.249 ms GcsInMemoryStore.Get - 572 total (0 active), Execution time: mean = 21.690 us, total = 12.407 ms, Queueing time: mean = 13.507 us, max = 64.163 us, min = 3.701 us, total = 7.726 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 571 total (0 active), Execution time: mean = 53.813 us, total = 30.727 ms, Queueing time: mean = 106.632 us, max = 269.544 us, min = 13.461 us, total = 60.887 ms InternalKVGcsService.grpc_server.InternalKVGet - 571 total (0 active), Execution time: mean = 592.337 us, total = 338.224 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 503 total (0 active), Execution time: mean = 91.837 us, total = 46.194 ms, Queueing time: mean = 116.363 us, max = 296.469 us, min = 19.828 us, total = 58.531 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 503 total (0 active), Execution time: mean = 681.641 us, total = 342.866 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 252 total (1 active), Execution time: mean = 1.425 ms, total = 359.149 ms, Queueing time: mean = 72.037 us, max = 315.299 us, min = -0.000 s, total = 18.153 ms GCSServer.deadline_timer.debug_state_event_stats_print - 42 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 65.435 ms, Queueing time: mean = 60.255 us, max = 108.649 us, min = 15.007 us, total = 2.531 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:43:53,873 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 116298 total (1 active) Queueing time: mean = 48.195 us, max = 7.565 ms, min = 2.216 us, total = 5.605 s Execution time: mean = 248.837 us, total = 28.939 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 52857 total (0 active), Execution time: mean = 515.412 us, total = 27.243 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 52857 total (0 active), Execution time: mean = 26.548 us, total = 1.403 s, Queueing time: mean = 103.275 us, max = 7.565 ms, min = 3.372 us, total = 5.459 s event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 28.498 us, total = 287.198 ms, Queueing time: mean = 10.476 us, max = 4.983 ms, min = 2.216 us, total = 105.575 ms GcsTaskManager.GcJobSummary - 505 total (1 active), Execution time: mean = 11.253 us, total = 5.683 ms, Queueing time: mean = 80.117 us, max = 229.756 us, min = 18.742 us, total = 40.459 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:43:53,873 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12486 total (25 active) Queueing time: mean = 19.316 us, max = 333.684 us, min = 2.330 us, total = 241.182 ms Execution time: mean = 4.592 s, total = 57339.960 s Event stats: event_loop_lag_probe - 10078 total (0 active), Execution time: mean = 30.849 us, total = 310.895 ms, Queueing time: mean = 11.327 us, max = 291.078 us, min = 2.330 us, total = 114.156 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 645 total (0 active), Execution time: mean = 42.738 us, total = 27.566 ms, Queueing time: mean = 102.237 us, max = 333.684 us, min = 8.944 us, total = 65.943 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 645 total (24 active), Execution time: mean = 88.898 s, total = 57339.202 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 529 total (0 active), Execution time: mean = 113.366 us, total = 59.970 ms, Queueing time: mean = 107.639 us, max = 247.252 us, min = 21.083 us, total = 56.941 ms InternalPubSubGcsService.grpc_server.GcsPublish - 529 total (0 active), Execution time: mean = 637.290 us, total = 337.126 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 680.274 us, total = 6.122 ms, Queueing time: mean = 80.517 us, max = 153.556 us, min = 48.361 us, total = 724.654 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:43:53,873 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10105 total (0 active) Queueing time: mean = 10.228 us, max = 1.423 ms, min = 568.000 ns, total = 103.356 ms Execution time: mean = 27.118 us, total = 274.023 ms Event stats: event_loop_lag_probe - 10079 total (0 active), Execution time: mean = 27.034 us, total = 272.474 ms, Queueing time: mean = 10.068 us, max = 1.423 ms, min = 1.850 us, total = 101.477 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:44:53,873 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 515 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 541 - cumulative published bytes: 426050 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:44:53,874 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 31335 total (4 active) Queueing time: mean = 122.063 us, max = 279.303 ms, min = -0.000 s, total = 3.825 s Execution time: mean = 246.906 us, total = 7.737 s Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 23.841 us, total = 245.992 ms, Queueing time: mean = 10.420 us, max = 13.097 ms, min = 2.088 us, total = 107.512 ms GcsInMemoryStore.Put - 4172 total (0 active), Execution time: mean = 72.450 us, total = 302.260 ms, Queueing time: mean = 513.092 us, max = 278.545 ms, min = 2.025 us, total = 2.141 s RayletLoadPulled - 2580 total (1 active), Execution time: mean = 244.231 us, total = 630.117 ms, Queueing time: mean = 74.687 us, max = 472.090 us, min = 11.570 us, total = 192.692 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2577 total (0 active), Execution time: mean = 76.820 us, total = 197.965 ms, Queueing time: mean = 105.463 us, max = 330.146 us, min = 11.910 us, total = 271.777 ms NodeManagerService.grpc_client.GetResourceLoad - 2577 total (0 active), Execution time: mean = 1.475 ms, total = 3.801 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1044 total (0 active), Execution time: mean = 41.693 us, total = 43.527 ms, Queueing time: mean = 103.780 us, max = 228.583 us, min = 11.443 us, total = 108.346 ms InternalKVGcsService.grpc_server.InternalKVPut - 1044 total (0 active), Execution time: mean = 558.433 us, total = 583.004 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1032 total (0 active), Execution time: mean = 33.848 us, total = 34.931 ms, Queueing time: mean = 118.149 us, max = 1.498 ms, min = 20.329 us, total = 121.930 ms NodeInfoGcsService.grpc_server.CheckAlive - 1032 total (0 active), Execution time: mean = 583.835 us, total = 602.518 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 861 total (1 active), Execution time: mean = 9.459 us, total = 8.144 ms, Queueing time: mean = 75.282 us, max = 260.630 us, min = 16.354 us, total = 64.818 ms HealthCheck - 858 total (0 active), Execution time: mean = 7.492 us, total = 6.428 ms, Queueing time: mean = 101.265 us, max = 399.856 us, min = 16.806 us, total = 86.885 ms GcsInMemoryStore.Get - 584 total (0 active), Execution time: mean = 21.667 us, total = 12.653 ms, Queueing time: mean = 13.484 us, max = 64.163 us, min = 3.701 us, total = 7.875 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 583 total (0 active), Execution time: mean = 53.707 us, total = 31.311 ms, Queueing time: mean = 106.636 us, max = 269.544 us, min = 13.461 us, total = 62.169 ms InternalKVGcsService.grpc_server.InternalKVGet - 583 total (0 active), Execution time: mean = 591.927 us, total = 345.094 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 515 total (0 active), Execution time: mean = 91.575 us, total = 47.161 ms, Queueing time: mean = 116.158 us, max = 296.469 us, min = 19.828 us, total = 59.821 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 515 total (0 active), Execution time: mean = 680.193 us, total = 350.299 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 258 total (1 active), Execution time: mean = 1.423 ms, total = 367.193 ms, Queueing time: mean = 71.411 us, max = 315.299 us, min = -0.000 s, total = 18.424 ms GCSServer.deadline_timer.debug_state_event_stats_print - 43 total (1 active, 1 running), Execution time: mean = 1.561 ms, total = 67.139 ms, Queueing time: mean = 60.301 us, max = 108.649 us, min = 15.007 us, total = 2.593 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:44:53,875 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 119070 total (1 active) Queueing time: mean = 48.239 us, max = 7.565 ms, min = 2.216 us, total = 5.744 s Execution time: mean = 248.979 us, total = 29.646 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 54117 total (0 active), Execution time: mean = 515.715 us, total = 27.909 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 54117 total (0 active), Execution time: mean = 26.544 us, total = 1.436 s, Queueing time: mean = 103.371 us, max = 7.565 ms, min = 3.372 us, total = 5.594 s event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 28.564 us, total = 294.723 ms, Queueing time: mean = 10.490 us, max = 4.983 ms, min = 2.216 us, total = 108.232 ms GcsTaskManager.GcJobSummary - 517 total (1 active), Execution time: mean = 11.246 us, total = 5.814 ms, Queueing time: mean = 80.037 us, max = 229.756 us, min = 18.742 us, total = 41.379 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:44:53,875 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 12774 total (25 active) Queueing time: mean = 19.293 us, max = 333.684 us, min = 2.330 us, total = 246.445 ms Execution time: mean = 4.494 s, total = 57400.105 s Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 30.890 us, total = 318.724 ms, Queueing time: mean = 11.346 us, max = 291.078 us, min = 2.330 us, total = 117.069 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 657 total (0 active), Execution time: mean = 42.939 us, total = 28.211 ms, Queueing time: mean = 102.046 us, max = 333.684 us, min = 8.944 us, total = 67.044 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 657 total (24 active), Execution time: mean = 87.366 s, total = 57399.330 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 541 total (0 active), Execution time: mean = 113.204 us, total = 61.244 ms, Queueing time: mean = 107.561 us, max = 247.252 us, min = 21.083 us, total = 58.190 ms InternalPubSubGcsService.grpc_server.GcsPublish - 541 total (0 active), Execution time: mean = 637.225 us, total = 344.739 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 680.274 us, total = 6.122 ms, Queueing time: mean = 80.517 us, max = 153.556 us, min = 48.361 us, total = 724.654 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:44:53,875 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10344 total (0 active) Queueing time: mean = 10.202 us, max = 1.423 ms, min = 568.000 ns, total = 105.532 ms Execution time: mean = 27.076 us, total = 280.073 ms Event stats: event_loop_lag_probe - 10318 total (0 active), Execution time: mean = 26.994 us, total = 278.524 ms, Queueing time: mean = 10.046 us, max = 1.423 ms, min = 1.850 us, total = 103.652 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:45:53,875 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 527 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 553 - cumulative published bytes: 435878 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:45:53,876 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 32054 total (4 active) Queueing time: mean = 120.272 us, max = 279.303 ms, min = -0.000 s, total = 3.855 s Execution time: mean = 246.919 us, total = 7.915 s Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 23.890 us, total = 252.232 ms, Queueing time: mean = 10.415 us, max = 13.097 ms, min = 2.088 us, total = 109.961 ms GcsInMemoryStore.Put - 4268 total (0 active), Execution time: mean = 70.924 us, total = 302.704 ms, Queueing time: mean = 502.544 us, max = 278.545 ms, min = 2.025 us, total = 2.145 s RayletLoadPulled - 2640 total (1 active), Execution time: mean = 244.719 us, total = 646.057 ms, Queueing time: mean = 74.756 us, max = 472.090 us, min = 11.570 us, total = 197.355 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2637 total (0 active), Execution time: mean = 76.861 us, total = 202.683 ms, Queueing time: mean = 105.614 us, max = 330.146 us, min = 11.910 us, total = 278.504 ms NodeManagerService.grpc_client.GetResourceLoad - 2637 total (0 active), Execution time: mean = 1.477 ms, total = 3.894 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1068 total (0 active), Execution time: mean = 41.714 us, total = 44.550 ms, Queueing time: mean = 103.674 us, max = 228.583 us, min = 11.443 us, total = 110.724 ms InternalKVGcsService.grpc_server.InternalKVPut - 1068 total (0 active), Execution time: mean = 558.883 us, total = 596.887 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1056 total (0 active), Execution time: mean = 33.802 us, total = 35.695 ms, Queueing time: mean = 118.089 us, max = 1.498 ms, min = 20.329 us, total = 124.702 ms NodeInfoGcsService.grpc_server.CheckAlive - 1056 total (0 active), Execution time: mean = 584.248 us, total = 616.966 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 881 total (1 active), Execution time: mean = 9.470 us, total = 8.343 ms, Queueing time: mean = 75.436 us, max = 260.630 us, min = 16.354 us, total = 66.459 ms HealthCheck - 878 total (0 active), Execution time: mean = 7.505 us, total = 6.590 ms, Queueing time: mean = 101.746 us, max = 399.856 us, min = 16.806 us, total = 89.333 ms GcsInMemoryStore.Get - 596 total (0 active), Execution time: mean = 21.652 us, total = 12.905 ms, Queueing time: mean = 13.462 us, max = 64.163 us, min = 3.701 us, total = 8.024 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 595 total (0 active), Execution time: mean = 54.005 us, total = 32.133 ms, Queueing time: mean = 106.304 us, max = 269.544 us, min = 13.461 us, total = 63.251 ms InternalKVGcsService.grpc_server.InternalKVGet - 595 total (0 active), Execution time: mean = 592.234 us, total = 352.379 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 527 total (0 active), Execution time: mean = 91.547 us, total = 48.245 ms, Queueing time: mean = 116.032 us, max = 296.469 us, min = 19.828 us, total = 61.149 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 527 total (0 active), Execution time: mean = 680.075 us, total = 358.400 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 264 total (1 active), Execution time: mean = 1.422 ms, total = 375.301 ms, Queueing time: mean = 71.393 us, max = 315.299 us, min = -0.000 s, total = 18.848 ms GCSServer.deadline_timer.debug_state_event_stats_print - 44 total (1 active, 1 running), Execution time: mean = 1.565 ms, total = 68.842 ms, Queueing time: mean = 60.468 us, max = 108.649 us, min = 15.007 us, total = 2.661 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:45:53,876 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 121842 total (1 active) Queueing time: mean = 48.307 us, max = 7.565 ms, min = 2.216 us, total = 5.886 s Execution time: mean = 249.389 us, total = 30.386 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 55377 total (0 active), Execution time: mean = 516.549 us, total = 28.605 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 55377 total (0 active), Execution time: mean = 26.603 us, total = 1.473 s, Queueing time: mean = 103.519 us, max = 7.565 ms, min = 3.372 us, total = 5.733 s event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 28.604 us, total = 302.004 ms, Queueing time: mean = 10.493 us, max = 4.983 ms, min = 2.216 us, total = 110.786 ms GcsTaskManager.GcJobSummary - 529 total (1 active), Execution time: mean = 11.220 us, total = 5.936 ms, Queueing time: mean = 80.062 us, max = 229.756 us, min = 18.742 us, total = 42.353 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:45:53,876 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13062 total (25 active) Queueing time: mean = 19.324 us, max = 333.684 us, min = 2.330 us, total = 252.413 ms Execution time: mean = 4.399 s, total = 57460.289 s Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 30.974 us, total = 327.028 ms, Queueing time: mean = 11.380 us, max = 291.078 us, min = 2.330 us, total = 120.154 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 669 total (0 active), Execution time: mean = 43.044 us, total = 28.796 ms, Queueing time: mean = 102.427 us, max = 333.684 us, min = 8.944 us, total = 68.523 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 669 total (24 active), Execution time: mean = 85.889 s, total = 57459.495 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 553 total (0 active), Execution time: mean = 113.185 us, total = 62.591 ms, Queueing time: mean = 107.764 us, max = 247.252 us, min = 21.083 us, total = 59.594 ms InternalPubSubGcsService.grpc_server.GcsPublish - 553 total (0 active), Execution time: mean = 637.663 us, total = 352.627 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 9 total (1 active), Execution time: mean = 680.274 us, total = 6.122 ms, Queueing time: mean = 80.517 us, max = 153.556 us, min = 48.361 us, total = 724.654 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:45:53,877 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10584 total (0 active) Queueing time: mean = 10.197 us, max = 1.423 ms, min = 568.000 ns, total = 107.921 ms Execution time: mean = 27.086 us, total = 286.683 ms Event stats: event_loop_lag_probe - 10558 total (0 active), Execution time: mean = 27.006 us, total = 285.134 ms, Queueing time: mean = 10.044 us, max = 1.423 ms, min = 1.850 us, total = 106.042 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:46:53,877 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 539 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 565 - cumulative published bytes: 445706 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:46:53,878 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 32773 total (4 active) Queueing time: mean = 118.512 us, max = 279.303 ms, min = -0.000 s, total = 3.884 s Execution time: mean = 246.821 us, total = 8.089 s Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 23.883 us, total = 257.893 ms, Queueing time: mean = 10.380 us, max = 13.097 ms, min = 2.088 us, total = 112.083 ms GcsInMemoryStore.Put - 4364 total (0 active), Execution time: mean = 69.473 us, total = 303.181 ms, Queueing time: mean = 492.363 us, max = 278.545 ms, min = 2.025 us, total = 2.149 s RayletLoadPulled - 2700 total (1 active), Execution time: mean = 244.913 us, total = 661.266 ms, Queueing time: mean = 74.835 us, max = 472.090 us, min = 11.570 us, total = 202.054 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2697 total (0 active), Execution time: mean = 76.891 us, total = 207.375 ms, Queueing time: mean = 105.232 us, max = 330.146 us, min = 11.910 us, total = 283.810 ms NodeManagerService.grpc_client.GetResourceLoad - 2697 total (0 active), Execution time: mean = 1.477 ms, total = 3.983 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1092 total (0 active), Execution time: mean = 42.047 us, total = 45.915 ms, Queueing time: mean = 103.872 us, max = 228.583 us, min = 11.443 us, total = 113.428 ms InternalKVGcsService.grpc_server.InternalKVPut - 1092 total (0 active), Execution time: mean = 559.866 us, total = 611.374 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1080 total (0 active), Execution time: mean = 33.789 us, total = 36.492 ms, Queueing time: mean = 118.015 us, max = 1.498 ms, min = 20.329 us, total = 127.456 ms NodeInfoGcsService.grpc_server.CheckAlive - 1080 total (0 active), Execution time: mean = 584.365 us, total = 631.114 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 901 total (1 active), Execution time: mean = 9.457 us, total = 8.521 ms, Queueing time: mean = 75.189 us, max = 260.630 us, min = 16.354 us, total = 67.745 ms HealthCheck - 898 total (0 active), Execution time: mean = 7.531 us, total = 6.763 ms, Queueing time: mean = 102.208 us, max = 399.856 us, min = 16.806 us, total = 91.782 ms GcsInMemoryStore.Get - 608 total (0 active), Execution time: mean = 21.630 us, total = 13.151 ms, Queueing time: mean = 13.502 us, max = 64.163 us, min = 3.701 us, total = 8.209 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 607 total (0 active), Execution time: mean = 54.141 us, total = 32.864 ms, Queueing time: mean = 106.618 us, max = 269.544 us, min = 13.461 us, total = 64.717 ms InternalKVGcsService.grpc_server.InternalKVGet - 607 total (0 active), Execution time: mean = 593.295 us, total = 360.130 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 539 total (0 active), Execution time: mean = 91.568 us, total = 49.355 ms, Queueing time: mean = 116.220 us, max = 296.469 us, min = 19.828 us, total = 62.643 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 539 total (0 active), Execution time: mean = 680.937 us, total = 367.025 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 270 total (1 active), Execution time: mean = 1.419 ms, total = 383.147 ms, Queueing time: mean = 71.041 us, max = 315.299 us, min = -0.000 s, total = 19.181 ms GCSServer.deadline_timer.debug_state_event_stats_print - 45 total (1 active, 1 running), Execution time: mean = 1.564 ms, total = 70.385 ms, Queueing time: mean = 63.169 us, max = 182.019 us, min = 15.007 us, total = 2.843 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:46:53,878 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 124614 total (1 active) Queueing time: mean = 48.358 us, max = 7.565 ms, min = 2.216 us, total = 6.026 s Execution time: mean = 249.534 us, total = 31.095 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 56637 total (0 active), Execution time: mean = 516.807 us, total = 29.270 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 56637 total (0 active), Execution time: mean = 26.647 us, total = 1.509 s, Queueing time: mean = 103.629 us, max = 7.565 ms, min = 3.372 us, total = 5.869 s event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 28.677 us, total = 309.659 ms, Queueing time: mean = 10.510 us, max = 4.983 ms, min = 2.216 us, total = 113.485 ms GcsTaskManager.GcJobSummary - 541 total (1 active), Execution time: mean = 11.328 us, total = 6.128 ms, Queueing time: mean = 80.073 us, max = 229.756 us, min = 18.742 us, total = 43.319 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:46:53,878 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13355 total (25 active) Queueing time: mean = 19.349 us, max = 333.684 us, min = 2.330 us, total = 258.407 ms Execution time: mean = 4.374 s, total = 58420.502 s Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 31.059 us, total = 335.380 ms, Queueing time: mean = 11.413 us, max = 291.078 us, min = 2.330 us, total = 123.238 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 683 total (0 active), Execution time: mean = 43.153 us, total = 29.473 ms, Queueing time: mean = 102.591 us, max = 333.684 us, min = 8.944 us, total = 70.070 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 683 total (24 active), Execution time: mean = 85.534 s, total = 58419.689 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 565 total (0 active), Execution time: mean = 113.493 us, total = 64.123 ms, Queueing time: mean = 107.768 us, max = 247.252 us, min = 21.083 us, total = 60.889 ms InternalPubSubGcsService.grpc_server.GcsPublish - 565 total (0 active), Execution time: mean = 638.421 us, total = 360.708 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 620.761 us, total = 6.208 ms, Queueing time: mean = 79.279 us, max = 153.556 us, min = 48.361 us, total = 792.794 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:46:53,878 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 10824 total (0 active) Queueing time: mean = 10.176 us, max = 1.423 ms, min = 568.000 ns, total = 110.148 ms Execution time: mean = 27.046 us, total = 292.743 ms Event stats: event_loop_lag_probe - 10798 total (0 active), Execution time: mean = 26.967 us, total = 291.193 ms, Queueing time: mean = 10.027 us, max = 1.423 ms, min = 1.850 us, total = 108.269 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:47:53,879 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 551 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 579 - cumulative published bytes: 455602 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:47:53,880 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 33492 total (4 active) Queueing time: mean = 116.908 us, max = 279.303 ms, min = -0.000 s, total = 3.915 s Execution time: mean = 246.732 us, total = 8.264 s Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 23.902 us, total = 263.827 ms, Queueing time: mean = 10.354 us, max = 13.097 ms, min = 2.088 us, total = 114.284 ms GcsInMemoryStore.Put - 4460 total (0 active), Execution time: mean = 68.086 us, total = 303.663 ms, Queueing time: mean = 483.237 us, max = 278.545 ms, min = 2.025 us, total = 2.155 s RayletLoadPulled - 2760 total (1 active), Execution time: mean = 245.031 us, total = 676.285 ms, Queueing time: mean = 74.862 us, max = 472.090 us, min = 11.570 us, total = 206.619 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2757 total (0 active), Execution time: mean = 76.858 us, total = 211.896 ms, Queueing time: mean = 105.053 us, max = 330.146 us, min = 11.910 us, total = 289.631 ms NodeManagerService.grpc_client.GetResourceLoad - 2757 total (0 active), Execution time: mean = 1.478 ms, total = 4.075 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1116 total (0 active), Execution time: mean = 41.978 us, total = 46.847 ms, Queueing time: mean = 104.079 us, max = 228.583 us, min = 11.443 us, total = 116.152 ms InternalKVGcsService.grpc_server.InternalKVPut - 1116 total (0 active), Execution time: mean = 560.001 us, total = 624.961 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1104 total (0 active), Execution time: mean = 33.793 us, total = 37.308 ms, Queueing time: mean = 117.948 us, max = 1.498 ms, min = 20.329 us, total = 130.215 ms NodeInfoGcsService.grpc_server.CheckAlive - 1104 total (0 active), Execution time: mean = 583.983 us, total = 644.717 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 921 total (1 active), Execution time: mean = 9.447 us, total = 8.701 ms, Queueing time: mean = 74.960 us, max = 260.630 us, min = 16.354 us, total = 69.038 ms HealthCheck - 918 total (0 active), Execution time: mean = 7.536 us, total = 6.918 ms, Queueing time: mean = 102.316 us, max = 399.856 us, min = 16.806 us, total = 93.926 ms GcsInMemoryStore.Get - 620 total (0 active), Execution time: mean = 21.632 us, total = 13.412 ms, Queueing time: mean = 13.486 us, max = 64.163 us, min = 3.701 us, total = 8.362 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 619 total (0 active), Execution time: mean = 54.143 us, total = 33.515 ms, Queueing time: mean = 106.597 us, max = 269.544 us, min = 13.461 us, total = 65.984 ms InternalKVGcsService.grpc_server.InternalKVGet - 619 total (0 active), Execution time: mean = 593.425 us, total = 367.330 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 551 total (0 active), Execution time: mean = 91.794 us, total = 50.579 ms, Queueing time: mean = 116.316 us, max = 296.469 us, min = 19.828 us, total = 64.090 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 551 total (0 active), Execution time: mean = 680.994 us, total = 375.228 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 276 total (1 active), Execution time: mean = 1.417 ms, total = 391.196 ms, Queueing time: mean = 71.238 us, max = 315.299 us, min = -0.000 s, total = 19.662 ms GCSServer.deadline_timer.debug_state_event_stats_print - 46 total (1 active, 1 running), Execution time: mean = 1.568 ms, total = 72.128 ms, Queueing time: mean = 63.141 us, max = 182.019 us, min = 15.007 us, total = 2.904 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:47:53,880 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 127386 total (1 active) Queueing time: mean = 48.440 us, max = 7.565 ms, min = 2.216 us, total = 6.171 s Execution time: mean = 249.837 us, total = 31.826 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 57897 total (0 active), Execution time: mean = 517.435 us, total = 29.958 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 57897 total (0 active), Execution time: mean = 26.679 us, total = 1.545 s, Queueing time: mean = 103.806 us, max = 7.565 ms, min = 3.372 us, total = 6.010 s event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 28.709 us, total = 316.886 ms, Queueing time: mean = 10.514 us, max = 4.983 ms, min = 2.216 us, total = 116.055 ms GcsTaskManager.GcJobSummary - 553 total (1 active), Execution time: mean = 11.346 us, total = 6.274 ms, Queueing time: mean = 80.144 us, max = 229.756 us, min = 18.742 us, total = 44.320 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:47:53,880 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13651 total (25 active) Queueing time: mean = 19.355 us, max = 333.684 us, min = 2.330 us, total = 264.211 ms Execution time: mean = 4.284 s, total = 58480.646 s Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 31.089 us, total = 343.161 ms, Queueing time: mean = 11.423 us, max = 291.078 us, min = 2.330 us, total = 126.090 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 697 total (0 active), Execution time: mean = 43.257 us, total = 30.150 ms, Queueing time: mean = 102.516 us, max = 333.684 us, min = 8.944 us, total = 71.454 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 697 total (24 active), Execution time: mean = 83.902 s, total = 58479.814 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 579 total (0 active), Execution time: mean = 113.652 us, total = 65.805 ms, Queueing time: mean = 107.871 us, max = 247.252 us, min = 21.083 us, total = 62.457 ms InternalPubSubGcsService.grpc_server.GcsPublish - 579 total (0 active), Execution time: mean = 638.783 us, total = 369.855 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 620.761 us, total = 6.208 ms, Queueing time: mean = 79.279 us, max = 153.556 us, min = 48.361 us, total = 792.794 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:47:53,880 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11064 total (0 active) Queueing time: mean = 10.158 us, max = 1.423 ms, min = 568.000 ns, total = 112.389 ms Execution time: mean = 27.015 us, total = 298.890 ms Event stats: event_loop_lag_probe - 11038 total (0 active), Execution time: mean = 26.938 us, total = 297.341 ms, Queueing time: mean = 10.012 us, max = 1.423 ms, min = 1.850 us, total = 110.510 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:48:53,880 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 563 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 591 - cumulative published bytes: 465430 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:48:53,881 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 34211 total (4 active) Queueing time: mean = 115.378 us, max = 279.303 ms, min = -0.000 s, total = 3.947 s Execution time: mean = 246.844 us, total = 8.445 s Event stats: event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 23.955 us, total = 270.170 ms, Queueing time: mean = 10.342 us, max = 13.097 ms, min = 2.088 us, total = 116.638 ms GcsInMemoryStore.Put - 4556 total (0 active), Execution time: mean = 66.760 us, total = 304.159 ms, Queueing time: mean = 474.112 us, max = 278.545 ms, min = 2.025 us, total = 2.160 s RayletLoadPulled - 2820 total (1 active), Execution time: mean = 245.394 us, total = 692.010 ms, Queueing time: mean = 75.086 us, max = 472.090 us, min = 11.570 us, total = 211.743 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2817 total (0 active), Execution time: mean = 76.965 us, total = 216.811 ms, Queueing time: mean = 105.258 us, max = 330.146 us, min = 11.910 us, total = 296.512 ms NodeManagerService.grpc_client.GetResourceLoad - 2817 total (0 active), Execution time: mean = 1.480 ms, total = 4.170 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1140 total (0 active), Execution time: mean = 41.853 us, total = 47.712 ms, Queueing time: mean = 103.779 us, max = 228.583 us, min = 11.443 us, total = 118.308 ms InternalKVGcsService.grpc_server.InternalKVPut - 1140 total (0 active), Execution time: mean = 559.901 us, total = 638.287 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1128 total (0 active), Execution time: mean = 33.911 us, total = 38.252 ms, Queueing time: mean = 118.045 us, max = 1.498 ms, min = 20.329 us, total = 133.155 ms NodeInfoGcsService.grpc_server.CheckAlive - 1128 total (0 active), Execution time: mean = 584.233 us, total = 659.014 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 941 total (1 active), Execution time: mean = 9.448 us, total = 8.891 ms, Queueing time: mean = 75.176 us, max = 260.630 us, min = 16.354 us, total = 70.741 ms HealthCheck - 938 total (0 active), Execution time: mean = 7.549 us, total = 7.081 ms, Queueing time: mean = 102.455 us, max = 399.856 us, min = 16.806 us, total = 96.103 ms GcsInMemoryStore.Get - 632 total (0 active), Execution time: mean = 21.637 us, total = 13.675 ms, Queueing time: mean = 13.482 us, max = 64.163 us, min = 3.701 us, total = 8.521 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 631 total (0 active), Execution time: mean = 54.112 us, total = 34.145 ms, Queueing time: mean = 106.638 us, max = 269.544 us, min = 13.461 us, total = 67.288 ms InternalKVGcsService.grpc_server.InternalKVGet - 631 total (0 active), Execution time: mean = 593.549 us, total = 374.529 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 563 total (0 active), Execution time: mean = 91.738 us, total = 51.648 ms, Queueing time: mean = 116.449 us, max = 296.469 us, min = 19.828 us, total = 65.561 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 563 total (0 active), Execution time: mean = 681.714 us, total = 383.805 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 282 total (1 active), Execution time: mean = 1.421 ms, total = 400.583 ms, Queueing time: mean = 71.642 us, max = 315.299 us, min = -0.000 s, total = 20.203 ms GCSServer.deadline_timer.debug_state_event_stats_print - 47 total (1 active, 1 running), Execution time: mean = 1.571 ms, total = 73.827 ms, Queueing time: mean = 63.647 us, max = 182.019 us, min = 15.007 us, total = 2.991 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:48:53,882 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 130153 total (1 active) Queueing time: mean = 48.520 us, max = 7.565 ms, min = 2.216 us, total = 6.315 s Execution time: mean = 250.115 us, total = 32.553 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 59155 total (0 active), Execution time: mean = 517.997 us, total = 30.642 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 59155 total (0 active), Execution time: mean = 26.722 us, total = 1.581 s, Queueing time: mean = 103.983 us, max = 7.565 ms, min = 3.372 us, total = 6.151 s event_loop_lag_probe - 11277 total (0 active), Execution time: mean = 28.722 us, total = 323.895 ms, Queueing time: mean = 10.508 us, max = 4.983 ms, min = 2.216 us, total = 118.503 ms GcsTaskManager.GcJobSummary - 565 total (1 active), Execution time: mean = 11.357 us, total = 6.417 ms, Queueing time: mean = 80.211 us, max = 229.756 us, min = 18.742 us, total = 45.319 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:48:53,882 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 13938 total (25 active) Queueing time: mean = 19.356 us, max = 333.684 us, min = 2.330 us, total = 269.788 ms Execution time: mean = 4.200 s, total = 58540.819 s Event stats: event_loop_lag_probe - 11277 total (0 active), Execution time: mean = 31.128 us, total = 351.031 ms, Queueing time: mean = 11.443 us, max = 291.078 us, min = 2.330 us, total = 129.041 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 709 total (0 active), Execution time: mean = 43.281 us, total = 30.686 ms, Queueing time: mean = 102.625 us, max = 333.684 us, min = 8.944 us, total = 72.761 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 709 total (24 active), Execution time: mean = 82.567 s, total = 58539.968 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 591 total (0 active), Execution time: mean = 114.536 us, total = 67.691 ms, Queueing time: mean = 107.912 us, max = 247.252 us, min = 21.083 us, total = 63.776 ms InternalPubSubGcsService.grpc_server.GcsPublish - 591 total (0 active), Execution time: mean = 639.718 us, total = 378.074 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 620.761 us, total = 6.208 ms, Queueing time: mean = 79.279 us, max = 153.556 us, min = 48.361 us, total = 792.794 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:48:53,882 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11304 total (0 active) Queueing time: mean = 10.158 us, max = 1.423 ms, min = 568.000 ns, total = 114.824 ms Execution time: mean = 27.022 us, total = 305.458 ms Event stats: event_loop_lag_probe - 11278 total (0 active), Execution time: mean = 26.947 us, total = 303.909 ms, Queueing time: mean = 10.015 us, max = 1.423 ms, min = 1.850 us, total = 112.944 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:49:53,882 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 575 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 604 - cumulative published bytes: 475292 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:49:53,883 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 34930 total (4 active) Queueing time: mean = 113.751 us, max = 279.303 ms, min = -0.000 s, total = 3.973 s Execution time: mean = 246.441 us, total = 8.608 s Event stats: event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 23.941 us, total = 275.753 ms, Queueing time: mean = 10.307 us, max = 13.097 ms, min = 2.088 us, total = 118.716 ms GcsInMemoryStore.Put - 4652 total (0 active), Execution time: mean = 65.483 us, total = 304.627 ms, Queueing time: mean = 465.209 us, max = 278.545 ms, min = 2.025 us, total = 2.164 s RayletLoadPulled - 2880 total (1 active), Execution time: mean = 245.006 us, total = 705.619 ms, Queueing time: mean = 74.911 us, max = 472.090 us, min = 11.570 us, total = 215.744 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2877 total (0 active), Execution time: mean = 76.876 us, total = 221.172 ms, Queueing time: mean = 105.068 us, max = 769.233 us, min = 11.910 us, total = 302.280 ms NodeManagerService.grpc_client.GetResourceLoad - 2877 total (0 active), Execution time: mean = 1.479 ms, total = 4.256 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1164 total (0 active), Execution time: mean = 41.849 us, total = 48.712 ms, Queueing time: mean = 103.527 us, max = 228.583 us, min = 11.443 us, total = 120.505 ms InternalKVGcsService.grpc_server.InternalKVPut - 1164 total (0 active), Execution time: mean = 559.387 us, total = 651.127 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1152 total (0 active), Execution time: mean = 33.845 us, total = 38.989 ms, Queueing time: mean = 117.575 us, max = 1.498 ms, min = 20.329 us, total = 135.446 ms NodeInfoGcsService.grpc_server.CheckAlive - 1152 total (0 active), Execution time: mean = 583.589 us, total = 672.295 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 961 total (1 active), Execution time: mean = 9.443 us, total = 9.075 ms, Queueing time: mean = 75.018 us, max = 260.630 us, min = 16.354 us, total = 72.092 ms HealthCheck - 958 total (0 active), Execution time: mean = 7.565 us, total = 7.247 ms, Queueing time: mean = 102.045 us, max = 399.856 us, min = 16.806 us, total = 97.759 ms GcsInMemoryStore.Get - 644 total (0 active), Execution time: mean = 21.559 us, total = 13.884 ms, Queueing time: mean = 13.454 us, max = 64.163 us, min = 3.701 us, total = 8.664 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 643 total (0 active), Execution time: mean = 54.335 us, total = 34.938 ms, Queueing time: mean = 106.187 us, max = 269.544 us, min = 13.461 us, total = 68.278 ms InternalKVGcsService.grpc_server.InternalKVGet - 643 total (0 active), Execution time: mean = 593.057 us, total = 381.336 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 575 total (0 active), Execution time: mean = 91.825 us, total = 52.799 ms, Queueing time: mean = 116.117 us, max = 296.469 us, min = 19.828 us, total = 66.768 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 575 total (0 active), Execution time: mean = 681.199 us, total = 391.689 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 288 total (1 active), Execution time: mean = 1.416 ms, total = 407.842 ms, Queueing time: mean = 71.223 us, max = 315.299 us, min = -0.000 s, total = 20.512 ms GCSServer.deadline_timer.debug_state_event_stats_print - 48 total (1 active, 1 running), Execution time: mean = 1.571 ms, total = 75.426 ms, Queueing time: mean = 63.026 us, max = 182.019 us, min = 15.007 us, total = 3.025 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:49:53,883 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 132925 total (1 active) Queueing time: mean = 48.524 us, max = 7.565 ms, min = 2.216 us, total = 6.450 s Execution time: mean = 250.108 us, total = 33.246 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 60415 total (0 active), Execution time: mean = 517.971 us, total = 31.293 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 60415 total (0 active), Execution time: mean = 26.729 us, total = 1.615 s, Queueing time: mean = 103.994 us, max = 7.565 ms, min = 3.372 us, total = 6.283 s event_loop_lag_probe - 11517 total (0 active), Execution time: mean = 28.740 us, total = 330.996 ms, Queueing time: mean = 10.507 us, max = 4.983 ms, min = 2.216 us, total = 121.007 ms GcsTaskManager.GcJobSummary - 577 total (1 active), Execution time: mean = 11.398 us, total = 6.577 ms, Queueing time: mean = 80.078 us, max = 229.756 us, min = 18.742 us, total = 46.205 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:49:53,883 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14230 total (25 active) Queueing time: mean = 19.313 us, max = 333.684 us, min = 2.330 us, total = 274.830 ms Execution time: mean = 4.118 s, total = 58601.009 s Event stats: event_loop_lag_probe - 11517 total (0 active), Execution time: mean = 31.127 us, total = 358.484 ms, Queueing time: mean = 11.444 us, max = 291.078 us, min = 2.330 us, total = 131.803 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 722 total (0 active), Execution time: mean = 43.242 us, total = 31.221 ms, Queueing time: mean = 102.231 us, max = 333.684 us, min = 8.944 us, total = 73.811 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 722 total (24 active), Execution time: mean = 81.164 s, total = 58600.140 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 604 total (0 active), Execution time: mean = 115.093 us, total = 69.516 ms, Queueing time: mean = 107.625 us, max = 247.252 us, min = 16.251 us, total = 65.006 ms InternalPubSubGcsService.grpc_server.GcsPublish - 604 total (0 active), Execution time: mean = 639.379 us, total = 386.185 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 620.761 us, total = 6.208 ms, Queueing time: mean = 79.279 us, max = 153.556 us, min = 48.361 us, total = 792.794 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:49:53,884 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11544 total (0 active) Queueing time: mean = 10.130 us, max = 1.423 ms, min = 568.000 ns, total = 116.941 ms Execution time: mean = 26.980 us, total = 311.457 ms Event stats: event_loop_lag_probe - 11518 total (0 active), Execution time: mean = 26.906 us, total = 309.908 ms, Queueing time: mean = 9.990 us, max = 1.423 ms, min = 1.850 us, total = 115.061 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:50:53,884 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 587 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 616 - cumulative published bytes: 485120 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:50:53,885 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 35649 total (4 active) Queueing time: mean = 112.180 us, max = 279.303 ms, min = -0.000 s, total = 3.999 s Execution time: mean = 246.129 us, total = 8.774 s Event stats: event_loop_lag_probe - 11758 total (0 active), Execution time: mean = 23.940 us, total = 281.491 ms, Queueing time: mean = 10.283 us, max = 13.097 ms, min = 2.088 us, total = 120.902 ms GcsInMemoryStore.Put - 4748 total (0 active), Execution time: mean = 64.266 us, total = 305.135 ms, Queueing time: mean = 456.714 us, max = 278.545 ms, min = 2.025 us, total = 2.168 s RayletLoadPulled - 2940 total (1 active), Execution time: mean = 245.285 us, total = 721.137 ms, Queueing time: mean = 74.697 us, max = 472.090 us, min = 11.570 us, total = 219.611 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2937 total (0 active), Execution time: mean = 76.787 us, total = 225.523 ms, Queueing time: mean = 104.804 us, max = 769.233 us, min = 11.910 us, total = 307.808 ms NodeManagerService.grpc_client.GetResourceLoad - 2937 total (0 active), Execution time: mean = 1.478 ms, total = 4.342 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1188 total (0 active), Execution time: mean = 41.757 us, total = 49.607 ms, Queueing time: mean = 103.309 us, max = 228.583 us, min = 11.443 us, total = 122.731 ms InternalKVGcsService.grpc_server.InternalKVPut - 1188 total (0 active), Execution time: mean = 559.180 us, total = 664.305 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1176 total (0 active), Execution time: mean = 33.822 us, total = 39.774 ms, Queueing time: mean = 117.027 us, max = 1.498 ms, min = 20.329 us, total = 137.624 ms NodeInfoGcsService.grpc_server.CheckAlive - 1176 total (0 active), Execution time: mean = 582.893 us, total = 685.482 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 981 total (1 active), Execution time: mean = 9.409 us, total = 9.230 ms, Queueing time: mean = 74.639 us, max = 260.630 us, min = 16.354 us, total = 73.221 ms HealthCheck - 978 total (0 active), Execution time: mean = 7.555 us, total = 7.389 ms, Queueing time: mean = 101.815 us, max = 399.856 us, min = 16.806 us, total = 99.575 ms GcsInMemoryStore.Get - 656 total (0 active), Execution time: mean = 21.484 us, total = 14.094 ms, Queueing time: mean = 13.398 us, max = 64.163 us, min = 3.701 us, total = 8.789 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 655 total (0 active), Execution time: mean = 54.206 us, total = 35.505 ms, Queueing time: mean = 105.626 us, max = 269.544 us, min = 13.461 us, total = 69.185 ms InternalKVGcsService.grpc_server.InternalKVGet - 655 total (0 active), Execution time: mean = 592.187 us, total = 387.882 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 587 total (0 active), Execution time: mean = 91.318 us, total = 53.604 ms, Queueing time: mean = 115.632 us, max = 296.469 us, min = 19.828 us, total = 67.876 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 587 total (0 active), Execution time: mean = 679.466 us, total = 398.847 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 294 total (1 active), Execution time: mean = 1.414 ms, total = 415.784 ms, Queueing time: mean = 70.921 us, max = 315.299 us, min = -0.000 s, total = 20.851 ms GCSServer.deadline_timer.debug_state_event_stats_print - 49 total (1 active, 1 running), Execution time: mean = 1.574 ms, total = 77.130 ms, Queueing time: mean = 62.902 us, max = 182.019 us, min = 15.007 us, total = 3.082 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:50:53,885 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 135697 total (1 active) Queueing time: mean = 48.555 us, max = 7.565 ms, min = 2.216 us, total = 6.589 s Execution time: mean = 250.206 us, total = 33.952 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 61675 total (0 active), Execution time: mean = 518.170 us, total = 31.958 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 61675 total (0 active), Execution time: mean = 26.742 us, total = 1.649 s, Queueing time: mean = 104.061 us, max = 7.565 ms, min = 3.372 us, total = 6.418 s event_loop_lag_probe - 11757 total (0 active), Execution time: mean = 28.750 us, total = 338.010 ms, Queueing time: mean = 10.501 us, max = 4.983 ms, min = 2.216 us, total = 123.464 ms GcsTaskManager.GcJobSummary - 589 total (1 active), Execution time: mean = 11.382 us, total = 6.704 ms, Queueing time: mean = 80.100 us, max = 229.756 us, min = 18.742 us, total = 47.179 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:50:53,885 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14518 total (25 active) Queueing time: mean = 19.324 us, max = 333.684 us, min = 2.330 us, total = 280.552 ms Execution time: mean = 4.041 s, total = 58661.214 s Event stats: event_loop_lag_probe - 11757 total (0 active), Execution time: mean = 31.161 us, total = 366.364 ms, Queueing time: mean = 11.458 us, max = 291.078 us, min = 2.330 us, total = 134.714 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 734 total (0 active), Execution time: mean = 43.245 us, total = 31.742 ms, Queueing time: mean = 102.380 us, max = 333.684 us, min = 8.944 us, total = 75.147 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 734 total (24 active), Execution time: mean = 79.919 s, total = 58660.328 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 616 total (0 active), Execution time: mean = 114.950 us, total = 70.809 ms, Queueing time: mean = 107.925 us, max = 247.252 us, min = 16.251 us, total = 66.482 ms InternalPubSubGcsService.grpc_server.GcsPublish - 616 total (0 active), Execution time: mean = 639.362 us, total = 393.847 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 10 total (1 active), Execution time: mean = 620.761 us, total = 6.208 ms, Queueing time: mean = 79.279 us, max = 153.556 us, min = 48.361 us, total = 792.794 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:50:53,885 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 11784 total (0 active) Queueing time: mean = 10.103 us, max = 1.423 ms, min = 568.000 ns, total = 119.055 ms Execution time: mean = 26.935 us, total = 317.408 ms Event stats: event_loop_lag_probe - 11758 total (0 active), Execution time: mean = 26.863 us, total = 315.859 ms, Queueing time: mean = 9.966 us, max = 1.423 ms, min = 1.850 us, total = 117.176 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:51:53,886 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 599 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 628 - cumulative published bytes: 494948 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:51:53,887 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 36368 total (4 active) Queueing time: mean = 110.854 us, max = 279.303 ms, min = -0.000 s, total = 4.032 s Execution time: mean = 246.331 us, total = 8.959 s Event stats: event_loop_lag_probe - 11998 total (0 active), Execution time: mean = 24.007 us, total = 288.038 ms, Queueing time: mean = 10.280 us, max = 13.097 ms, min = 2.088 us, total = 123.345 ms GcsInMemoryStore.Put - 4844 total (0 active), Execution time: mean = 63.111 us, total = 305.709 ms, Queueing time: mean = 448.695 us, max = 278.545 ms, min = 2.025 us, total = 2.173 s RayletLoadPulled - 3000 total (1 active), Execution time: mean = 245.673 us, total = 737.018 ms, Queueing time: mean = 74.891 us, max = 472.090 us, min = 11.570 us, total = 224.672 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 2997 total (0 active), Execution time: mean = 76.946 us, total = 230.606 ms, Queueing time: mean = 105.001 us, max = 769.233 us, min = 11.910 us, total = 314.687 ms NodeManagerService.grpc_client.GetResourceLoad - 2997 total (0 active), Execution time: mean = 1.480 ms, total = 4.436 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1212 total (0 active), Execution time: mean = 41.877 us, total = 50.755 ms, Queueing time: mean = 103.618 us, max = 228.583 us, min = 11.443 us, total = 125.585 ms InternalKVGcsService.grpc_server.InternalKVPut - 1212 total (0 active), Execution time: mean = 559.933 us, total = 678.638 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1200 total (0 active), Execution time: mean = 33.973 us, total = 40.768 ms, Queueing time: mean = 117.149 us, max = 1.498 ms, min = 20.329 us, total = 140.579 ms NodeInfoGcsService.grpc_server.CheckAlive - 1200 total (0 active), Execution time: mean = 583.962 us, total = 700.754 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1001 total (1 active), Execution time: mean = 9.559 us, total = 9.568 ms, Queueing time: mean = 74.495 us, max = 260.630 us, min = 16.354 us, total = 74.570 ms HealthCheck - 998 total (0 active), Execution time: mean = 7.563 us, total = 7.548 ms, Queueing time: mean = 102.105 us, max = 399.856 us, min = 16.806 us, total = 101.900 ms GcsInMemoryStore.Get - 668 total (0 active), Execution time: mean = 21.524 us, total = 14.378 ms, Queueing time: mean = 13.411 us, max = 64.163 us, min = 3.701 us, total = 8.958 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 667 total (0 active), Execution time: mean = 54.405 us, total = 36.288 ms, Queueing time: mean = 105.821 us, max = 269.544 us, min = 13.461 us, total = 70.583 ms InternalKVGcsService.grpc_server.InternalKVGet - 667 total (0 active), Execution time: mean = 593.344 us, total = 395.761 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 599 total (0 active), Execution time: mean = 91.482 us, total = 54.798 ms, Queueing time: mean = 115.805 us, max = 296.469 us, min = 19.828 us, total = 69.367 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 599 total (0 active), Execution time: mean = 680.190 us, total = 407.434 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 300 total (1 active), Execution time: mean = 1.417 ms, total = 425.190 ms, Queueing time: mean = 70.865 us, max = 315.299 us, min = -0.000 s, total = 21.259 ms GCSServer.deadline_timer.debug_state_event_stats_print - 50 total (1 active, 1 running), Execution time: mean = 1.575 ms, total = 78.729 ms, Queueing time: mean = 63.307 us, max = 182.019 us, min = 15.007 us, total = 3.165 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:51:53,887 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 138469 total (1 active) Queueing time: mean = 48.562 us, max = 7.565 ms, min = 2.216 us, total = 6.724 s Execution time: mean = 250.402 us, total = 34.673 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 62935 total (0 active), Execution time: mean = 518.544 us, total = 32.635 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 62935 total (0 active), Execution time: mean = 26.784 us, total = 1.686 s, Queueing time: mean = 104.073 us, max = 7.565 ms, min = 3.372 us, total = 6.550 s event_loop_lag_probe - 11997 total (0 active), Execution time: mean = 28.823 us, total = 345.786 ms, Queueing time: mean = 10.519 us, max = 4.983 ms, min = 2.216 us, total = 126.192 ms GcsTaskManager.GcJobSummary - 601 total (1 active), Execution time: mean = 11.428 us, total = 6.868 ms, Queueing time: mean = 80.299 us, max = 229.756 us, min = 18.742 us, total = 48.259 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:51:53,887 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 14849 total (25 active) Queueing time: mean = 19.434 us, max = 333.684 us, min = 2.330 us, total = 288.580 ms Execution time: mean = 4.803 s, total = 71321.399 s Event stats: event_loop_lag_probe - 11997 total (0 active), Execution time: mean = 31.241 us, total = 374.804 ms, Queueing time: mean = 11.486 us, max = 291.078 us, min = 2.330 us, total = 137.801 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 767 total (0 active), Execution time: mean = 42.626 us, total = 32.694 ms, Queueing time: mean = 102.493 us, max = 333.684 us, min = 8.944 us, total = 78.612 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 767 total (24 active), Execution time: mean = 92.986 s, total = 71320.493 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 628 total (0 active), Execution time: mean = 114.946 us, total = 72.186 ms, Queueing time: mean = 108.107 us, max = 247.252 us, min = 16.251 us, total = 67.891 ms InternalPubSubGcsService.grpc_server.GcsPublish - 628 total (0 active), Execution time: mean = 639.933 us, total = 401.878 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 653.022 us, total = 7.183 ms, Queueing time: mean = 78.108 us, max = 153.556 us, min = 48.361 us, total = 859.193 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:51:53,887 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12024 total (0 active) Queueing time: mean = 10.113 us, max = 1.423 ms, min = 568.000 ns, total = 121.597 ms Execution time: mean = 26.984 us, total = 324.456 ms Event stats: event_loop_lag_probe - 11998 total (0 active), Execution time: mean = 26.913 us, total = 322.907 ms, Queueing time: mean = 9.978 us, max = 1.423 ms, min = 1.850 us, total = 119.718 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:52:53,887 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 611 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 640 - cumulative published bytes: 504776 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:52:53,888 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 37086 total (4 active) Queueing time: mean = 109.604 us, max = 279.303 ms, min = -0.000 s, total = 4.065 s Execution time: mean = 246.578 us, total = 9.145 s Event stats: event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 24.082 us, total = 294.696 ms, Queueing time: mean = 10.283 us, max = 13.097 ms, min = 2.088 us, total = 125.838 ms GcsInMemoryStore.Put - 4940 total (0 active), Execution time: mean = 62.036 us, total = 306.460 ms, Queueing time: mean = 440.988 us, max = 278.545 ms, min = 2.025 us, total = 2.178 s RayletLoadPulled - 3060 total (1 active), Execution time: mean = 245.998 us, total = 752.755 ms, Queueing time: mean = 75.013 us, max = 472.090 us, min = 11.570 us, total = 229.540 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3057 total (0 active), Execution time: mean = 77.030 us, total = 235.480 ms, Queueing time: mean = 105.255 us, max = 769.233 us, min = 11.910 us, total = 321.764 ms NodeManagerService.grpc_client.GetResourceLoad - 3057 total (0 active), Execution time: mean = 1.482 ms, total = 4.532 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1236 total (0 active), Execution time: mean = 41.958 us, total = 51.860 ms, Queueing time: mean = 103.792 us, max = 228.583 us, min = 11.443 us, total = 128.287 ms InternalKVGcsService.grpc_server.InternalKVPut - 1236 total (0 active), Execution time: mean = 561.002 us, total = 693.399 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1224 total (0 active), Execution time: mean = 34.052 us, total = 41.679 ms, Queueing time: mean = 117.387 us, max = 1.498 ms, min = 20.329 us, total = 143.682 ms NodeInfoGcsService.grpc_server.CheckAlive - 1224 total (0 active), Execution time: mean = 584.821 us, total = 715.820 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1021 total (1 active), Execution time: mean = 9.576 us, total = 9.777 ms, Queueing time: mean = 74.795 us, max = 260.630 us, min = 16.354 us, total = 76.366 ms HealthCheck - 1018 total (0 active), Execution time: mean = 7.591 us, total = 7.727 ms, Queueing time: mean = 102.334 us, max = 399.856 us, min = 16.806 us, total = 104.176 ms GcsInMemoryStore.Get - 680 total (0 active), Execution time: mean = 21.489 us, total = 14.613 ms, Queueing time: mean = 13.414 us, max = 64.163 us, min = 3.701 us, total = 9.122 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 679 total (0 active), Execution time: mean = 54.678 us, total = 37.127 ms, Queueing time: mean = 106.030 us, max = 269.544 us, min = 13.461 us, total = 71.995 ms InternalKVGcsService.grpc_server.InternalKVGet - 679 total (0 active), Execution time: mean = 594.610 us, total = 403.740 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 611 total (0 active), Execution time: mean = 91.686 us, total = 56.020 ms, Queueing time: mean = 116.273 us, max = 296.469 us, min = 19.828 us, total = 71.043 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 611 total (0 active), Execution time: mean = 681.775 us, total = 416.564 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 306 total (1 active), Execution time: mean = 1.421 ms, total = 434.783 ms, Queueing time: mean = 71.260 us, max = 315.299 us, min = -0.000 s, total = 21.806 ms GCSServer.deadline_timer.debug_state_event_stats_print - 51 total (1 active, 1 running), Execution time: mean = 1.578 ms, total = 80.460 ms, Queueing time: mean = 64.599 us, max = 182.019 us, min = 15.007 us, total = 3.295 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:52:53,889 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 141241 total (1 active) Queueing time: mean = 48.611 us, max = 7.565 ms, min = 2.216 us, total = 6.866 s Execution time: mean = 250.685 us, total = 35.407 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 64195 total (0 active), Execution time: mean = 519.089 us, total = 33.323 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 64195 total (0 active), Execution time: mean = 26.862 us, total = 1.724 s, Queueing time: mean = 104.182 us, max = 7.565 ms, min = 3.372 us, total = 6.688 s event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 28.817 us, total = 352.638 ms, Queueing time: mean = 10.505 us, max = 4.983 ms, min = 2.216 us, total = 128.546 ms GcsTaskManager.GcJobSummary - 613 total (1 active), Execution time: mean = 11.405 us, total = 6.991 ms, Queueing time: mean = 80.408 us, max = 229.756 us, min = 18.742 us, total = 49.290 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:52:53,889 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15137 total (25 active) Queueing time: mean = 19.411 us, max = 333.684 us, min = 2.330 us, total = 293.818 ms Execution time: mean = 4.716 s, total = 71381.613 s Event stats: event_loop_lag_probe - 12237 total (0 active), Execution time: mean = 31.274 us, total = 382.696 ms, Queueing time: mean = 11.486 us, max = 291.078 us, min = 2.330 us, total = 140.554 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 779 total (0 active), Execution time: mean = 42.707 us, total = 33.269 ms, Queueing time: mean = 102.343 us, max = 333.684 us, min = 8.944 us, total = 79.725 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 779 total (24 active), Execution time: mean = 91.631 s, total = 71380.689 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 640 total (0 active), Execution time: mean = 115.103 us, total = 73.666 ms, Queueing time: mean = 108.222 us, max = 247.252 us, min = 16.251 us, total = 69.262 ms InternalPubSubGcsService.grpc_server.GcsPublish - 640 total (0 active), Execution time: mean = 641.662 us, total = 410.663 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 653.022 us, total = 7.183 ms, Queueing time: mean = 78.108 us, max = 153.556 us, min = 48.361 us, total = 859.193 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:52:53,889 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12264 total (0 active) Queueing time: mean = 10.127 us, max = 1.423 ms, min = 568.000 ns, total = 124.202 ms Execution time: mean = 27.023 us, total = 331.415 ms Event stats: event_loop_lag_probe - 12238 total (0 active), Execution time: mean = 26.954 us, total = 329.866 ms, Queueing time: mean = 9.995 us, max = 1.423 ms, min = 1.850 us, total = 122.323 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:53:53,889 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 623 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 652 - cumulative published bytes: 514604 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:53:53,890 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 37805 total (4 active) Queueing time: mean = 108.293 us, max = 279.303 ms, min = -0.000 s, total = 4.094 s Execution time: mean = 246.674 us, total = 9.326 s Event stats: event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 24.099 us, total = 300.684 ms, Queueing time: mean = 10.265 us, max = 13.097 ms, min = 2.088 us, total = 128.071 ms GcsInMemoryStore.Put - 5036 total (0 active), Execution time: mean = 60.952 us, total = 306.956 ms, Queueing time: mean = 433.527 us, max = 278.545 ms, min = 2.025 us, total = 2.183 s RayletLoadPulled - 3120 total (1 active), Execution time: mean = 246.127 us, total = 767.916 ms, Queueing time: mean = 74.937 us, max = 472.090 us, min = -0.000 s, total = 233.804 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3117 total (0 active), Execution time: mean = 77.061 us, total = 240.198 ms, Queueing time: mean = 105.103 us, max = 769.233 us, min = 11.910 us, total = 327.606 ms NodeManagerService.grpc_client.GetResourceLoad - 3117 total (0 active), Execution time: mean = 1.483 ms, total = 4.624 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1260 total (0 active), Execution time: mean = 42.126 us, total = 53.079 ms, Queueing time: mean = 103.682 us, max = 228.583 us, min = 11.443 us, total = 130.639 ms InternalKVGcsService.grpc_server.InternalKVPut - 1260 total (0 active), Execution time: mean = 561.569 us, total = 707.577 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1248 total (0 active), Execution time: mean = 34.260 us, total = 42.757 ms, Queueing time: mean = 117.286 us, max = 1.498 ms, min = 20.329 us, total = 146.373 ms NodeInfoGcsService.grpc_server.CheckAlive - 1248 total (0 active), Execution time: mean = 585.735 us, total = 730.997 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1041 total (1 active), Execution time: mean = 9.558 us, total = 9.950 ms, Queueing time: mean = 74.911 us, max = 260.630 us, min = 16.354 us, total = 77.983 ms HealthCheck - 1038 total (0 active), Execution time: mean = 7.600 us, total = 7.889 ms, Queueing time: mean = 102.143 us, max = 399.856 us, min = 16.806 us, total = 106.024 ms GcsInMemoryStore.Get - 692 total (0 active), Execution time: mean = 21.460 us, total = 14.850 ms, Queueing time: mean = 13.401 us, max = 64.163 us, min = 3.701 us, total = 9.273 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 691 total (0 active), Execution time: mean = 55.029 us, total = 38.025 ms, Queueing time: mean = 105.982 us, max = 269.544 us, min = 13.461 us, total = 73.234 ms InternalKVGcsService.grpc_server.InternalKVGet - 691 total (0 active), Execution time: mean = 594.955 us, total = 411.114 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 623 total (0 active), Execution time: mean = 92.215 us, total = 57.450 ms, Queueing time: mean = 116.735 us, max = 296.469 us, min = 19.828 us, total = 72.726 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 623 total (0 active), Execution time: mean = 683.655 us, total = 425.917 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 312 total (1 active), Execution time: mean = 1.423 ms, total = 443.939 ms, Queueing time: mean = 71.317 us, max = 315.299 us, min = -0.000 s, total = 22.251 ms GCSServer.deadline_timer.debug_state_event_stats_print - 52 total (1 active, 1 running), Execution time: mean = 1.581 ms, total = 82.203 ms, Queueing time: mean = 65.902 us, max = 182.019 us, min = 15.007 us, total = 3.427 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:53:53,890 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 144013 total (1 active) Queueing time: mean = 48.672 us, max = 7.565 ms, min = 2.216 us, total = 7.009 s Execution time: mean = 250.931 us, total = 36.137 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 65455 total (0 active), Execution time: mean = 519.572 us, total = 34.009 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 65455 total (0 active), Execution time: mean = 26.916 us, total = 1.762 s, Queueing time: mean = 104.317 us, max = 7.565 ms, min = 3.372 us, total = 6.828 s event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 28.846 us, total = 359.907 ms, Queueing time: mean = 10.499 us, max = 4.983 ms, min = 2.216 us, total = 130.994 ms GcsTaskManager.GcJobSummary - 625 total (1 active), Execution time: mean = 11.393 us, total = 7.120 ms, Queueing time: mean = 80.416 us, max = 229.756 us, min = 18.742 us, total = 50.260 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:53:53,891 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15425 total (25 active) Queueing time: mean = 19.407 us, max = 333.684 us, min = 2.330 us, total = 299.352 ms Execution time: mean = 4.632 s, total = 71441.763 s Event stats: event_loop_lag_probe - 12477 total (0 active), Execution time: mean = 31.301 us, total = 390.549 ms, Queueing time: mean = 11.499 us, max = 291.078 us, min = 2.330 us, total = 143.471 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 791 total (0 active), Execution time: mean = 42.852 us, total = 33.896 ms, Queueing time: mean = 102.352 us, max = 333.684 us, min = 8.944 us, total = 80.961 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 791 total (24 active), Execution time: mean = 90.317 s, total = 71440.821 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 652 total (0 active), Execution time: mean = 115.173 us, total = 75.093 ms, Queueing time: mean = 108.349 us, max = 247.252 us, min = 16.251 us, total = 70.644 ms InternalPubSubGcsService.grpc_server.GcsPublish - 652 total (0 active), Execution time: mean = 642.072 us, total = 418.631 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 653.022 us, total = 7.183 ms, Queueing time: mean = 78.108 us, max = 153.556 us, min = 48.361 us, total = 859.193 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:53:53,891 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12504 total (0 active) Queueing time: mean = 10.119 us, max = 1.423 ms, min = 568.000 ns, total = 126.526 ms Execution time: mean = 27.019 us, total = 337.846 ms Event stats: event_loop_lag_probe - 12478 total (0 active), Execution time: mean = 26.951 us, total = 336.297 ms, Queueing time: mean = 9.989 us, max = 1.423 ms, min = 1.850 us, total = 124.647 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:54:53,891 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 635 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 664 - cumulative published bytes: 524432 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:54:53,892 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 38524 total (4 active) Queueing time: mean = 107.150 us, max = 279.303 ms, min = -0.000 s, total = 4.128 s Execution time: mean = 246.822 us, total = 9.509 s Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 24.164 us, total = 307.299 ms, Queueing time: mean = 10.265 us, max = 13.097 ms, min = 2.088 us, total = 130.545 ms GcsInMemoryStore.Put - 5132 total (0 active), Execution time: mean = 59.916 us, total = 307.490 ms, Queueing time: mean = 426.399 us, max = 278.545 ms, min = 2.025 us, total = 2.188 s RayletLoadPulled - 3180 total (1 active), Execution time: mean = 246.625 us, total = 784.266 ms, Queueing time: mean = 75.130 us, max = 472.090 us, min = -0.000 s, total = 238.915 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3177 total (0 active), Execution time: mean = 77.203 us, total = 245.272 ms, Queueing time: mean = 105.451 us, max = 769.233 us, min = 11.910 us, total = 335.016 ms NodeManagerService.grpc_client.GetResourceLoad - 3177 total (0 active), Execution time: mean = 1.485 ms, total = 4.716 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1284 total (0 active), Execution time: mean = 42.280 us, total = 54.288 ms, Queueing time: mean = 103.943 us, max = 228.583 us, min = 11.443 us, total = 133.463 ms InternalKVGcsService.grpc_server.InternalKVPut - 1284 total (0 active), Execution time: mean = 562.164 us, total = 721.819 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1272 total (0 active), Execution time: mean = 34.358 us, total = 43.704 ms, Queueing time: mean = 117.578 us, max = 1.498 ms, min = 20.329 us, total = 149.559 ms NodeInfoGcsService.grpc_server.CheckAlive - 1272 total (0 active), Execution time: mean = 586.841 us, total = 746.461 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1061 total (1 active), Execution time: mean = 9.573 us, total = 10.157 ms, Queueing time: mean = 75.080 us, max = 260.630 us, min = 16.354 us, total = 79.660 ms HealthCheck - 1058 total (0 active), Execution time: mean = 7.623 us, total = 8.065 ms, Queueing time: mean = 102.463 us, max = 399.856 us, min = 16.806 us, total = 108.406 ms GcsInMemoryStore.Get - 704 total (0 active), Execution time: mean = 21.418 us, total = 15.078 ms, Queueing time: mean = 13.398 us, max = 64.163 us, min = 3.701 us, total = 9.432 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 703 total (0 active), Execution time: mean = 55.114 us, total = 38.745 ms, Queueing time: mean = 106.233 us, max = 269.544 us, min = 13.461 us, total = 74.682 ms InternalKVGcsService.grpc_server.InternalKVGet - 703 total (0 active), Execution time: mean = 595.400 us, total = 418.566 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 635 total (0 active), Execution time: mean = 92.716 us, total = 58.875 ms, Queueing time: mean = 117.061 us, max = 296.469 us, min = 19.828 us, total = 74.334 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 635 total (0 active), Execution time: mean = 685.204 us, total = 435.105 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 318 total (1 active), Execution time: mean = 1.424 ms, total = 452.761 ms, Queueing time: mean = 71.260 us, max = 315.299 us, min = -0.000 s, total = 22.661 ms GCSServer.deadline_timer.debug_state_event_stats_print - 53 total (1 active, 1 running), Execution time: mean = 1.585 ms, total = 84.014 ms, Queueing time: mean = 65.999 us, max = 182.019 us, min = 15.007 us, total = 3.498 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:54:53,892 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 146781 total (1 active) Queueing time: mean = 48.690 us, max = 7.565 ms, min = 2.216 us, total = 7.147 s Execution time: mean = 251.194 us, total = 36.870 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 66713 total (0 active), Execution time: mean = 520.088 us, total = 34.697 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 66713 total (0 active), Execution time: mean = 26.973 us, total = 1.799 s, Queueing time: mean = 104.360 us, max = 7.565 ms, min = 3.372 us, total = 6.962 s event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 28.869 us, total = 367.131 ms, Queueing time: mean = 10.491 us, max = 4.983 ms, min = 2.216 us, total = 133.411 ms GcsTaskManager.GcJobSummary - 637 total (1 active), Execution time: mean = 11.395 us, total = 7.259 ms, Queueing time: mean = 80.288 us, max = 229.756 us, min = 17.567 us, total = 51.144 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:54:53,893 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 15713 total (25 active) Queueing time: mean = 19.398 us, max = 333.684 us, min = 2.330 us, total = 304.800 ms Execution time: mean = 4.550 s, total = 71501.924 s Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 31.330 us, total = 398.422 ms, Queueing time: mean = 11.510 us, max = 291.078 us, min = 2.330 us, total = 146.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 803 total (0 active), Execution time: mean = 42.908 us, total = 34.455 ms, Queueing time: mean = 102.361 us, max = 333.684 us, min = 8.944 us, total = 82.196 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 803 total (24 active), Execution time: mean = 89.042 s, total = 71500.963 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 664 total (0 active), Execution time: mean = 115.816 us, total = 76.902 ms, Queueing time: mean = 108.360 us, max = 247.252 us, min = 16.251 us, total = 71.951 ms InternalPubSubGcsService.grpc_server.GcsPublish - 664 total (0 active), Execution time: mean = 643.097 us, total = 427.016 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 653.022 us, total = 7.183 ms, Queueing time: mean = 78.108 us, max = 153.556 us, min = 48.361 us, total = 859.193 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:54:53,893 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12743 total (0 active) Queueing time: mean = 10.124 us, max = 1.423 ms, min = 568.000 ns, total = 129.012 ms Execution time: mean = 27.076 us, total = 345.033 ms Event stats: event_loop_lag_probe - 12717 total (0 active), Execution time: mean = 27.010 us, total = 343.484 ms, Queueing time: mean = 9.997 us, max = 1.423 ms, min = 1.850 us, total = 127.133 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:55:53,893 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 647 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 676 - cumulative published bytes: 534260 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:55:53,894 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 39243 total (4 active) Queueing time: mean = 106.007 us, max = 279.303 ms, min = -0.000 s, total = 4.160 s Execution time: mean = 246.933 us, total = 9.690 s Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 24.233 us, total = 313.987 ms, Queueing time: mean = 10.273 us, max = 13.097 ms, min = 2.088 us, total = 133.110 ms GcsInMemoryStore.Put - 5228 total (0 active), Execution time: mean = 58.918 us, total = 308.026 ms, Queueing time: mean = 419.428 us, max = 278.545 ms, min = 2.025 us, total = 2.193 s RayletLoadPulled - 3240 total (1 active), Execution time: mean = 246.893 us, total = 799.934 ms, Queueing time: mean = 75.258 us, max = 472.090 us, min = -0.000 s, total = 243.837 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3237 total (0 active), Execution time: mean = 77.236 us, total = 250.012 ms, Queueing time: mean = 105.503 us, max = 769.233 us, min = 11.910 us, total = 341.515 ms NodeManagerService.grpc_client.GetResourceLoad - 3237 total (0 active), Execution time: mean = 1.485 ms, total = 4.808 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1308 total (0 active), Execution time: mean = 42.470 us, total = 55.551 ms, Queueing time: mean = 104.257 us, max = 228.583 us, min = 11.443 us, total = 136.368 ms InternalKVGcsService.grpc_server.InternalKVPut - 1308 total (0 active), Execution time: mean = 563.531 us, total = 737.099 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1296 total (0 active), Execution time: mean = 34.386 us, total = 44.565 ms, Queueing time: mean = 117.707 us, max = 1.498 ms, min = 20.329 us, total = 152.548 ms NodeInfoGcsService.grpc_server.CheckAlive - 1296 total (0 active), Execution time: mean = 587.459 us, total = 761.347 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1081 total (1 active), Execution time: mean = 9.611 us, total = 10.390 ms, Queueing time: mean = 75.363 us, max = 260.630 us, min = 16.354 us, total = 81.467 ms HealthCheck - 1078 total (0 active), Execution time: mean = 7.641 us, total = 8.236 ms, Queueing time: mean = 102.581 us, max = 399.856 us, min = 16.806 us, total = 110.582 ms GcsInMemoryStore.Get - 716 total (0 active), Execution time: mean = 21.397 us, total = 15.320 ms, Queueing time: mean = 13.410 us, max = 64.163 us, min = 3.701 us, total = 9.602 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 715 total (0 active), Execution time: mean = 55.129 us, total = 39.417 ms, Queueing time: mean = 106.651 us, max = 269.544 us, min = 13.461 us, total = 76.256 ms InternalKVGcsService.grpc_server.InternalKVGet - 715 total (0 active), Execution time: mean = 596.457 us, total = 426.467 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 647 total (0 active), Execution time: mean = 92.709 us, total = 59.983 ms, Queueing time: mean = 117.259 us, max = 296.469 us, min = 19.828 us, total = 75.867 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 647 total (0 active), Execution time: mean = 685.945 us, total = 443.807 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 324 total (1 active), Execution time: mean = 1.426 ms, total = 462.119 ms, Queueing time: mean = 71.338 us, max = 315.299 us, min = -0.000 s, total = 23.113 ms GCSServer.deadline_timer.debug_state_event_stats_print - 54 total (1 active, 1 running), Execution time: mean = 1.588 ms, total = 85.755 ms, Queueing time: mean = 67.151 us, max = 182.019 us, min = 15.007 us, total = 3.626 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:55:53,894 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 149551 total (1 active) Queueing time: mean = 48.718 us, max = 7.565 ms, min = 2.216 us, total = 7.286 s Execution time: mean = 251.430 us, total = 37.602 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 67972 total (0 active), Execution time: mean = 520.550 us, total = 35.383 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 67972 total (0 active), Execution time: mean = 27.026 us, total = 1.837 s, Queueing time: mean = 104.420 us, max = 7.565 ms, min = 3.372 us, total = 7.098 s event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 28.895 us, total = 374.399 ms, Queueing time: mean = 10.493 us, max = 4.983 ms, min = 2.216 us, total = 135.960 ms GcsTaskManager.GcJobSummary - 649 total (1 active), Execution time: mean = 11.413 us, total = 7.407 ms, Queueing time: mean = 80.211 us, max = 229.756 us, min = 17.567 us, total = 52.057 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:55:53,894 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16001 total (25 active) Queueing time: mean = 19.395 us, max = 333.684 us, min = 2.330 us, total = 310.332 ms Execution time: mean = 4.472 s, total = 71562.120 s Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 31.353 us, total = 406.237 ms, Queueing time: mean = 11.514 us, max = 291.078 us, min = 2.330 us, total = 149.193 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 815 total (0 active), Execution time: mean = 43.038 us, total = 35.076 ms, Queueing time: mean = 102.456 us, max = 333.684 us, min = 8.944 us, total = 83.502 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 815 total (24 active), Execution time: mean = 87.805 s, total = 71561.141 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 676 total (0 active), Execution time: mean = 116.178 us, total = 78.536 ms, Queueing time: mean = 108.522 us, max = 247.252 us, min = 16.251 us, total = 73.361 ms InternalPubSubGcsService.grpc_server.GcsPublish - 676 total (0 active), Execution time: mean = 643.857 us, total = 435.247 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 11 total (1 active), Execution time: mean = 653.022 us, total = 7.183 ms, Queueing time: mean = 78.108 us, max = 153.556 us, min = 48.361 us, total = 859.193 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:55:53,894 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 12983 total (0 active) Queueing time: mean = 10.139 us, max = 1.423 ms, min = 568.000 ns, total = 131.640 ms Execution time: mean = 27.130 us, total = 352.231 ms Event stats: event_loop_lag_probe - 12957 total (0 active), Execution time: mean = 27.065 us, total = 350.682 ms, Queueing time: mean = 10.015 us, max = 1.423 ms, min = 1.850 us, total = 129.760 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:56:53,895 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 658 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 688 - cumulative published bytes: 543303 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:56:53,895 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 39951 total (4 active) Queueing time: mean = 104.876 us, max = 279.303 ms, min = -0.000 s, total = 4.190 s Execution time: mean = 246.911 us, total = 9.864 s Event stats: event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 24.244 us, total = 319.948 ms, Queueing time: mean = 10.256 us, max = 13.097 ms, min = 2.088 us, total = 135.347 ms GcsInMemoryStore.Put - 5322 total (0 active), Execution time: mean = 57.960 us, total = 308.465 ms, Queueing time: mean = 412.849 us, max = 278.545 ms, min = 2.025 us, total = 2.197 s RayletLoadPulled - 3300 total (1 active), Execution time: mean = 247.091 us, total = 815.399 ms, Queueing time: mean = 75.309 us, max = 472.090 us, min = -0.000 s, total = 248.519 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3297 total (0 active), Execution time: mean = 77.223 us, total = 254.605 ms, Queueing time: mean = 105.548 us, max = 769.233 us, min = 11.910 us, total = 347.992 ms NodeManagerService.grpc_client.GetResourceLoad - 3297 total (0 active), Execution time: mean = 1.486 ms, total = 4.900 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1330 total (0 active), Execution time: mean = 42.456 us, total = 56.467 ms, Queueing time: mean = 104.461 us, max = 228.583 us, min = 11.443 us, total = 138.933 ms InternalKVGcsService.grpc_server.InternalKVPut - 1330 total (0 active), Execution time: mean = 564.060 us, total = 750.200 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1320 total (0 active), Execution time: mean = 34.353 us, total = 45.346 ms, Queueing time: mean = 117.665 us, max = 1.498 ms, min = 20.329 us, total = 155.318 ms NodeInfoGcsService.grpc_server.CheckAlive - 1320 total (0 active), Execution time: mean = 588.029 us, total = 776.199 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1101 total (1 active), Execution time: mean = 9.614 us, total = 10.585 ms, Queueing time: mean = 75.241 us, max = 260.630 us, min = 16.354 us, total = 82.841 ms HealthCheck - 1098 total (0 active), Execution time: mean = 7.666 us, total = 8.417 ms, Queueing time: mean = 102.540 us, max = 399.856 us, min = 16.806 us, total = 112.589 ms GcsInMemoryStore.Get - 727 total (0 active), Execution time: mean = 21.608 us, total = 15.709 ms, Queueing time: mean = 13.421 us, max = 64.163 us, min = 3.701 us, total = 9.757 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 726 total (0 active), Execution time: mean = 55.211 us, total = 40.083 ms, Queueing time: mean = 106.753 us, max = 269.544 us, min = 13.461 us, total = 77.503 ms InternalKVGcsService.grpc_server.InternalKVGet - 726 total (0 active), Execution time: mean = 597.376 us, total = 433.695 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 658 total (0 active), Execution time: mean = 92.491 us, total = 60.859 ms, Queueing time: mean = 117.525 us, max = 296.469 us, min = 19.828 us, total = 77.332 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 658 total (0 active), Execution time: mean = 685.718 us, total = 451.203 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 330 total (1 active), Execution time: mean = 1.425 ms, total = 470.148 ms, Queueing time: mean = 71.189 us, max = 315.299 us, min = -0.000 s, total = 23.492 ms GCSServer.deadline_timer.debug_state_event_stats_print - 55 total (1 active, 1 running), Execution time: mean = 1.589 ms, total = 87.417 ms, Queueing time: mean = 67.484 us, max = 182.019 us, min = 15.007 us, total = 3.712 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:56:53,895 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 152322 total (1 active) Queueing time: mean = 48.794 us, max = 7.565 ms, min = 2.216 us, total = 7.432 s Execution time: mean = 251.658 us, total = 38.333 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 69232 total (0 active), Execution time: mean = 521.003 us, total = 36.070 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 69232 total (0 active), Execution time: mean = 27.071 us, total = 1.874 s, Queueing time: mean = 104.590 us, max = 7.565 ms, min = 3.372 us, total = 7.241 s event_loop_lag_probe - 13196 total (0 active), Execution time: mean = 28.884 us, total = 381.159 ms, Queueing time: mean = 10.481 us, max = 4.983 ms, min = 2.216 us, total = 138.307 ms GcsTaskManager.GcJobSummary - 661 total (1 active), Execution time: mean = 11.425 us, total = 7.552 ms, Queueing time: mean = 80.232 us, max = 229.756 us, min = 17.567 us, total = 53.033 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:56:53,895 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16293 total (25 active) Queueing time: mean = 19.430 us, max = 333.684 us, min = 2.330 us, total = 316.569 ms Execution time: mean = 4.469 s, total = 72817.364 s Event stats: event_loop_lag_probe - 13196 total (0 active), Execution time: mean = 31.388 us, total = 414.194 ms, Queueing time: mean = 11.530 us, max = 291.078 us, min = 2.330 us, total = 152.148 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 829 total (0 active), Execution time: mean = 43.123 us, total = 35.749 ms, Queueing time: mean = 102.923 us, max = 333.684 us, min = 8.944 us, total = 85.324 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 829 total (24 active), Execution time: mean = 87.836 s, total = 72816.367 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 688 total (0 active), Execution time: mean = 116.123 us, total = 79.893 ms, Queueing time: mean = 108.657 us, max = 247.252 us, min = 16.251 us, total = 74.756 ms InternalPubSubGcsService.grpc_server.GcsPublish - 688 total (0 active), Execution time: mean = 643.921 us, total = 443.018 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 603.938 us, total = 7.247 ms, Queueing time: mean = 77.051 us, max = 153.556 us, min = 48.361 us, total = 924.607 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:56:53,895 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13223 total (0 active) Queueing time: mean = 10.126 us, max = 1.423 ms, min = 568.000 ns, total = 133.898 ms Execution time: mean = 27.111 us, total = 358.486 ms Event stats: event_loop_lag_probe - 13197 total (0 active), Execution time: mean = 27.047 us, total = 356.937 ms, Queueing time: mean = 10.004 us, max = 1.423 ms, min = 1.850 us, total = 132.019 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:57:53,896 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 670 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 701 - cumulative published bytes: 553165 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:57:53,896 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 40670 total (4 active) Queueing time: mean = 103.654 us, max = 279.303 ms, min = -0.000 s, total = 4.216 s Execution time: mean = 246.676 us, total = 10.032 s Event stats: event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 24.251 us, total = 325.855 ms, Queueing time: mean = 10.238 us, max = 13.097 ms, min = 2.088 us, total = 137.570 ms GcsInMemoryStore.Put - 5418 total (0 active), Execution time: mean = 57.014 us, total = 308.902 ms, Queueing time: mean = 406.349 us, max = 278.545 ms, min = 2.025 us, total = 2.202 s RayletLoadPulled - 3360 total (1 active), Execution time: mean = 247.110 us, total = 830.290 ms, Queueing time: mean = 75.145 us, max = 472.090 us, min = -0.000 s, total = 252.486 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3357 total (0 active), Execution time: mean = 77.166 us, total = 259.045 ms, Queueing time: mean = 105.130 us, max = 769.233 us, min = 11.910 us, total = 352.920 ms NodeManagerService.grpc_client.GetResourceLoad - 3357 total (0 active), Execution time: mean = 1.486 ms, total = 4.989 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1354 total (0 active), Execution time: mean = 42.432 us, total = 57.453 ms, Queueing time: mean = 103.932 us, max = 228.583 us, min = 11.443 us, total = 140.725 ms InternalKVGcsService.grpc_server.InternalKVPut - 1354 total (0 active), Execution time: mean = 563.735 us, total = 763.298 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1344 total (0 active), Execution time: mean = 34.461 us, total = 46.316 ms, Queueing time: mean = 117.398 us, max = 1.498 ms, min = 20.329 us, total = 157.782 ms NodeInfoGcsService.grpc_server.CheckAlive - 1344 total (0 active), Execution time: mean = 587.922 us, total = 790.168 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1121 total (1 active), Execution time: mean = 9.594 us, total = 10.755 ms, Queueing time: mean = 75.333 us, max = 260.630 us, min = 16.354 us, total = 84.449 ms HealthCheck - 1118 total (0 active), Execution time: mean = 7.663 us, total = 8.567 ms, Queueing time: mean = 102.457 us, max = 399.856 us, min = 16.806 us, total = 114.546 ms GcsInMemoryStore.Get - 739 total (0 active), Execution time: mean = 21.524 us, total = 15.906 ms, Queueing time: mean = 13.383 us, max = 64.163 us, min = 3.701 us, total = 9.890 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 738 total (0 active), Execution time: mean = 55.060 us, total = 40.634 ms, Queueing time: mean = 106.226 us, max = 269.544 us, min = 13.461 us, total = 78.395 ms InternalKVGcsService.grpc_server.InternalKVGet - 738 total (0 active), Execution time: mean = 596.341 us, total = 440.099 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 670 total (0 active), Execution time: mean = 92.114 us, total = 61.717 ms, Queueing time: mean = 116.838 us, max = 296.469 us, min = 19.828 us, total = 78.282 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 670 total (0 active), Execution time: mean = 684.405 us, total = 458.551 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 336 total (1 active), Execution time: mean = 1.422 ms, total = 477.761 ms, Queueing time: mean = 70.932 us, max = 315.299 us, min = -0.000 s, total = 23.833 ms GCSServer.deadline_timer.debug_state_event_stats_print - 56 total (1 active, 1 running), Execution time: mean = 1.578 ms, total = 88.385 ms, Queueing time: mean = 66.715 us, max = 182.019 us, min = 15.007 us, total = 3.736 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:57:53,896 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 155092 total (1 active) Queueing time: mean = 48.822 us, max = 7.565 ms, min = 2.216 us, total = 7.572 s Execution time: mean = 251.738 us, total = 39.043 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 70491 total (0 active), Execution time: mean = 521.160 us, total = 36.737 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 70491 total (0 active), Execution time: mean = 27.097 us, total = 1.910 s, Queueing time: mean = 104.653 us, max = 7.565 ms, min = 3.372 us, total = 7.377 s event_loop_lag_probe - 13436 total (0 active), Execution time: mean = 28.852 us, total = 387.655 ms, Queueing time: mean = 10.461 us, max = 4.983 ms, min = 2.216 us, total = 140.559 ms GcsTaskManager.GcJobSummary - 673 total (1 active), Execution time: mean = 11.474 us, total = 7.722 ms, Queueing time: mean = 80.336 us, max = 229.756 us, min = 17.567 us, total = 54.066 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:57:53,897 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16585 total (25 active) Queueing time: mean = 19.436 us, max = 333.684 us, min = 2.330 us, total = 322.342 ms Execution time: mean = 4.394 s, total = 72877.452 s Event stats: event_loop_lag_probe - 13436 total (0 active), Execution time: mean = 31.388 us, total = 421.734 ms, Queueing time: mean = 11.534 us, max = 291.078 us, min = 2.330 us, total = 154.977 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 842 total (0 active), Execution time: mean = 43.072 us, total = 36.267 ms, Queueing time: mean = 103.105 us, max = 333.684 us, min = 8.944 us, total = 86.815 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 842 total (24 active), Execution time: mean = 86.552 s, total = 72876.437 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 701 total (0 active), Execution time: mean = 116.007 us, total = 81.321 ms, Queueing time: mean = 108.713 us, max = 247.252 us, min = 16.251 us, total = 76.208 ms InternalPubSubGcsService.grpc_server.GcsPublish - 701 total (0 active), Execution time: mean = 644.247 us, total = 451.617 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 603.938 us, total = 7.247 ms, Queueing time: mean = 77.051 us, max = 153.556 us, min = 48.361 us, total = 924.607 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:57:53,897 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13463 total (0 active) Queueing time: mean = 10.101 us, max = 1.423 ms, min = 568.000 ns, total = 135.986 ms Execution time: mean = 27.051 us, total = 364.183 ms Event stats: event_loop_lag_probe - 13437 total (0 active), Execution time: mean = 26.988 us, total = 362.634 ms, Queueing time: mean = 9.980 us, max = 1.423 ms, min = 1.850 us, total = 134.107 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:58:53,897 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 682 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 713 - cumulative published bytes: 562993 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:58:53,898 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 41389 total (4 active) Queueing time: mean = 102.631 us, max = 279.303 ms, min = -0.000 s, total = 4.248 s Execution time: mean = 246.805 us, total = 10.215 s Event stats: event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 24.284 us, total = 332.137 ms, Queueing time: mean = 10.229 us, max = 13.097 ms, min = 2.088 us, total = 139.899 ms GcsInMemoryStore.Put - 5514 total (0 active), Execution time: mean = 56.121 us, total = 309.452 ms, Queueing time: mean = 400.092 us, max = 278.545 ms, min = 2.025 us, total = 2.206 s RayletLoadPulled - 3420 total (1 active), Execution time: mean = 247.211 us, total = 845.463 ms, Queueing time: mean = 75.159 us, max = 472.090 us, min = -0.000 s, total = 257.045 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3417 total (0 active), Execution time: mean = 77.196 us, total = 263.777 ms, Queueing time: mean = 105.291 us, max = 769.233 us, min = 11.910 us, total = 359.779 ms NodeManagerService.grpc_client.GetResourceLoad - 3417 total (0 active), Execution time: mean = 1.488 ms, total = 5.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1378 total (0 active), Execution time: mean = 42.610 us, total = 58.716 ms, Queueing time: mean = 104.076 us, max = 228.583 us, min = 11.443 us, total = 143.417 ms InternalKVGcsService.grpc_server.InternalKVPut - 1378 total (0 active), Execution time: mean = 564.404 us, total = 777.749 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1368 total (0 active), Execution time: mean = 34.507 us, total = 47.206 ms, Queueing time: mean = 117.505 us, max = 1.498 ms, min = 20.329 us, total = 160.747 ms NodeInfoGcsService.grpc_server.CheckAlive - 1368 total (0 active), Execution time: mean = 588.397 us, total = 804.927 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1141 total (1 active), Execution time: mean = 9.584 us, total = 10.935 ms, Queueing time: mean = 75.907 us, max = 328.578 us, min = 16.354 us, total = 86.609 ms HealthCheck - 1138 total (0 active), Execution time: mean = 7.676 us, total = 8.736 ms, Queueing time: mean = 102.732 us, max = 399.856 us, min = 16.806 us, total = 116.909 ms GcsInMemoryStore.Get - 751 total (0 active), Execution time: mean = 21.542 us, total = 16.178 ms, Queueing time: mean = 13.408 us, max = 64.163 us, min = 3.701 us, total = 10.069 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 750 total (0 active), Execution time: mean = 55.531 us, total = 41.648 ms, Queueing time: mean = 106.491 us, max = 269.544 us, min = 13.461 us, total = 79.868 ms InternalKVGcsService.grpc_server.InternalKVGet - 750 total (0 active), Execution time: mean = 596.748 us, total = 447.561 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 682 total (0 active), Execution time: mean = 92.500 us, total = 63.085 ms, Queueing time: mean = 117.063 us, max = 296.469 us, min = 19.828 us, total = 79.837 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 682 total (0 active), Execution time: mean = 685.257 us, total = 467.345 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 342 total (1 active), Execution time: mean = 1.422 ms, total = 486.433 ms, Queueing time: mean = 71.238 us, max = 315.299 us, min = -0.000 s, total = 24.363 ms GCSServer.deadline_timer.debug_state_event_stats_print - 57 total (1 active, 1 running), Execution time: mean = 1.570 ms, total = 89.464 ms, Queueing time: mean = 66.152 us, max = 182.019 us, min = 15.007 us, total = 3.771 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:58:53,898 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 157846 total (1 active) Queueing time: mean = 48.859 us, max = 7.565 ms, min = 2.216 us, total = 7.712 s Execution time: mean = 251.937 us, total = 39.767 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 71742 total (0 active), Execution time: mean = 521.562 us, total = 37.418 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 71742 total (0 active), Execution time: mean = 27.135 us, total = 1.947 s, Queueing time: mean = 104.737 us, max = 7.565 ms, min = 3.372 us, total = 7.514 s event_loop_lag_probe - 13676 total (0 active), Execution time: mean = 28.861 us, total = 394.705 ms, Queueing time: mean = 10.454 us, max = 4.983 ms, min = 2.216 us, total = 142.973 ms GcsTaskManager.GcJobSummary - 685 total (1 active), Execution time: mean = 11.485 us, total = 7.867 ms, Queueing time: mean = 80.380 us, max = 229.756 us, min = 17.567 us, total = 55.060 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:58:53,898 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 16873 total (25 active) Queueing time: mean = 19.448 us, max = 333.684 us, min = 2.330 us, total = 328.148 ms Execution time: mean = 4.323 s, total = 72937.681 s Event stats: event_loop_lag_probe - 13676 total (0 active), Execution time: mean = 31.410 us, total = 429.561 ms, Queueing time: mean = 11.548 us, max = 291.078 us, min = 2.330 us, total = 157.935 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 854 total (0 active), Execution time: mean = 43.197 us, total = 36.890 ms, Queueing time: mean = 103.216 us, max = 333.684 us, min = 8.944 us, total = 88.146 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 854 total (24 active), Execution time: mean = 85.406 s, total = 72936.647 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 713 total (0 active), Execution time: mean = 116.021 us, total = 82.723 ms, Queueing time: mean = 109.009 us, max = 247.252 us, min = 16.251 us, total = 77.724 ms InternalPubSubGcsService.grpc_server.GcsPublish - 713 total (0 active), Execution time: mean = 645.157 us, total = 459.997 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 603.938 us, total = 7.247 ms, Queueing time: mean = 77.051 us, max = 153.556 us, min = 48.361 us, total = 924.607 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:58:53,898 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13703 total (0 active) Queueing time: mean = 10.102 us, max = 1.423 ms, min = 568.000 ns, total = 138.434 ms Execution time: mean = 27.064 us, total = 370.858 ms Event stats: event_loop_lag_probe - 13677 total (0 active), Execution time: mean = 27.002 us, total = 369.309 ms, Queueing time: mean = 9.984 us, max = 1.423 ms, min = 1.850 us, total = 136.555 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-20 23:59:53,898 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 694 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 725 - cumulative published bytes: 572821 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-20 23:59:53,899 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 42105 total (4 active) Queueing time: mean = 101.632 us, max = 279.303 ms, min = -0.000 s, total = 4.279 s Execution time: mean = 246.734 us, total = 10.389 s Event stats: event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 24.361 us, total = 339.029 ms, Queueing time: mean = 10.240 us, max = 13.097 ms, min = 2.088 us, total = 142.509 ms GcsInMemoryStore.Put - 5610 total (0 active), Execution time: mean = 55.261 us, total = 310.016 ms, Queueing time: mean = 394.305 us, max = 278.545 ms, min = 2.025 us, total = 2.212 s RayletLoadPulled - 3479 total (1 active), Execution time: mean = 247.527 us, total = 861.145 ms, Queueing time: mean = 75.194 us, max = 472.090 us, min = -0.000 s, total = 261.600 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3476 total (0 active), Execution time: mean = 77.192 us, total = 268.319 ms, Queueing time: mean = 105.182 us, max = 769.233 us, min = 11.910 us, total = 365.614 ms NodeManagerService.grpc_client.GetResourceLoad - 3476 total (0 active), Execution time: mean = 1.488 ms, total = 5.172 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1402 total (0 active), Execution time: mean = 42.520 us, total = 59.613 ms, Queueing time: mean = 104.132 us, max = 228.583 us, min = 11.443 us, total = 145.994 ms InternalKVGcsService.grpc_server.InternalKVPut - 1402 total (0 active), Execution time: mean = 564.278 us, total = 791.118 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1392 total (0 active), Execution time: mean = 34.547 us, total = 48.089 ms, Queueing time: mean = 117.614 us, max = 1.498 ms, min = 20.329 us, total = 163.718 ms NodeInfoGcsService.grpc_server.CheckAlive - 1392 total (0 active), Execution time: mean = 588.724 us, total = 819.503 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1161 total (1 active), Execution time: mean = 9.600 us, total = 11.146 ms, Queueing time: mean = 75.822 us, max = 328.578 us, min = 16.354 us, total = 88.030 ms HealthCheck - 1158 total (0 active), Execution time: mean = 7.687 us, total = 8.901 ms, Queueing time: mean = 102.837 us, max = 399.856 us, min = 16.806 us, total = 119.085 ms GcsInMemoryStore.Get - 763 total (0 active), Execution time: mean = 21.534 us, total = 16.430 ms, Queueing time: mean = 13.392 us, max = 64.163 us, min = 3.701 us, total = 10.218 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 762 total (0 active), Execution time: mean = 55.478 us, total = 42.274 ms, Queueing time: mean = 106.203 us, max = 269.544 us, min = 13.461 us, total = 80.927 ms InternalKVGcsService.grpc_server.InternalKVGet - 762 total (0 active), Execution time: mean = 596.684 us, total = 454.673 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 694 total (0 active), Execution time: mean = 92.419 us, total = 64.139 ms, Queueing time: mean = 117.290 us, max = 296.469 us, min = 19.828 us, total = 81.399 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 694 total (0 active), Execution time: mean = 685.214 us, total = 475.538 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 348 total (1 active), Execution time: mean = 1.424 ms, total = 495.497 ms, Queueing time: mean = 71.447 us, max = 315.299 us, min = -0.000 s, total = 24.864 ms GCSServer.deadline_timer.debug_state_event_stats_print - 58 total (1 active, 1 running), Execution time: mean = 1.571 ms, total = 91.114 ms, Queueing time: mean = 66.168 us, max = 182.019 us, min = 15.007 us, total = 3.838 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-20 23:59:53,900 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 160606 total (1 active) Queueing time: mean = 48.900 us, max = 7.565 ms, min = 2.216 us, total = 7.854 s Execution time: mean = 252.060 us, total = 40.482 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 72996 total (0 active), Execution time: mean = 521.806 us, total = 38.090 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 72996 total (0 active), Execution time: mean = 27.160 us, total = 1.983 s, Queueing time: mean = 104.827 us, max = 7.565 ms, min = 3.372 us, total = 7.652 s event_loop_lag_probe - 13916 total (0 active), Execution time: mean = 28.889 us, total = 402.020 ms, Queueing time: mean = 10.457 us, max = 4.983 ms, min = 2.216 us, total = 145.522 ms GcsTaskManager.GcJobSummary - 697 total (1 active), Execution time: mean = 11.493 us, total = 8.010 ms, Queueing time: mean = 80.537 us, max = 229.756 us, min = 17.567 us, total = 56.135 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-20 23:59:53,900 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17161 total (25 active) Queueing time: mean = 19.441 us, max = 333.684 us, min = 2.330 us, total = 333.625 ms Execution time: mean = 4.254 s, total = 72997.792 s Event stats: event_loop_lag_probe - 13916 total (0 active), Execution time: mean = 31.417 us, total = 437.199 ms, Queueing time: mean = 11.550 us, max = 291.078 us, min = 2.330 us, total = 160.731 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 866 total (0 active), Execution time: mean = 43.339 us, total = 37.532 ms, Queueing time: mean = 103.441 us, max = 333.684 us, min = 8.944 us, total = 89.580 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 866 total (24 active), Execution time: mean = 84.292 s, total = 72996.741 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 725 total (0 active), Execution time: mean = 115.971 us, total = 84.079 ms, Queueing time: mean = 108.927 us, max = 247.252 us, min = 16.251 us, total = 78.972 ms InternalPubSubGcsService.grpc_server.GcsPublish - 725 total (0 active), Execution time: mean = 645.705 us, total = 468.136 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 603.938 us, total = 7.247 ms, Queueing time: mean = 77.051 us, max = 153.556 us, min = 48.361 us, total = 924.607 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-20 23:59:53,900 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 13943 total (0 active) Queueing time: mean = 10.096 us, max = 1.423 ms, min = 568.000 ns, total = 140.764 ms Execution time: mean = 27.055 us, total = 377.228 ms Event stats: event_loop_lag_probe - 13917 total (0 active), Execution time: mean = 26.994 us, total = 375.679 ms, Queueing time: mean = 9.980 us, max = 1.423 ms, min = 1.850 us, total = 138.885 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:00:53,900 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 706 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 737 - cumulative published bytes: 582649 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:00:53,901 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 42824 total (4 active) Queueing time: mean = 100.525 us, max = 279.303 ms, min = -0.000 s, total = 4.305 s Execution time: mean = 246.732 us, total = 10.566 s Event stats: event_loop_lag_probe - 14157 total (0 active), Execution time: mean = 24.349 us, total = 344.708 ms, Queueing time: mean = 10.216 us, max = 13.097 ms, min = 2.088 us, total = 144.628 ms GcsInMemoryStore.Put - 5706 total (0 active), Execution time: mean = 54.412 us, total = 310.473 ms, Queueing time: mean = 388.388 us, max = 278.545 ms, min = 2.025 us, total = 2.216 s RayletLoadPulled - 3539 total (1 active), Execution time: mean = 248.727 us, total = 880.245 ms, Queueing time: mean = 75.014 us, max = 472.090 us, min = -0.000 s, total = 265.475 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3536 total (0 active), Execution time: mean = 77.183 us, total = 272.918 ms, Queueing time: mean = 104.868 us, max = 769.233 us, min = 11.910 us, total = 370.812 ms NodeManagerService.grpc_client.GetResourceLoad - 3536 total (0 active), Execution time: mean = 1.488 ms, total = 5.263 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1426 total (0 active), Execution time: mean = 42.439 us, total = 60.518 ms, Queueing time: mean = 103.788 us, max = 228.583 us, min = 11.443 us, total = 148.002 ms InternalKVGcsService.grpc_server.InternalKVPut - 1426 total (0 active), Execution time: mean = 563.806 us, total = 803.987 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1416 total (0 active), Execution time: mean = 34.784 us, total = 49.254 ms, Queueing time: mean = 117.571 us, max = 1.498 ms, min = 20.329 us, total = 166.480 ms NodeInfoGcsService.grpc_server.CheckAlive - 1416 total (0 active), Execution time: mean = 588.933 us, total = 833.929 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1181 total (1 active), Execution time: mean = 9.580 us, total = 11.314 ms, Queueing time: mean = 75.593 us, max = 328.578 us, min = 16.354 us, total = 89.275 ms HealthCheck - 1178 total (0 active), Execution time: mean = 7.670 us, total = 9.035 ms, Queueing time: mean = 102.343 us, max = 399.856 us, min = 16.806 us, total = 120.560 ms GcsInMemoryStore.Get - 775 total (0 active), Execution time: mean = 21.518 us, total = 16.677 ms, Queueing time: mean = 13.370 us, max = 64.163 us, min = 3.701 us, total = 10.362 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 774 total (0 active), Execution time: mean = 55.460 us, total = 42.926 ms, Queueing time: mean = 105.962 us, max = 269.544 us, min = 13.461 us, total = 82.015 ms InternalKVGcsService.grpc_server.InternalKVGet - 774 total (0 active), Execution time: mean = 596.427 us, total = 461.635 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 706 total (0 active), Execution time: mean = 92.394 us, total = 65.230 ms, Queueing time: mean = 117.007 us, max = 296.469 us, min = 19.828 us, total = 82.607 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 706 total (0 active), Execution time: mean = 685.058 us, total = 483.651 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 354 total (1 active), Execution time: mean = 1.423 ms, total = 503.681 ms, Queueing time: mean = 71.451 us, max = 315.299 us, min = -0.000 s, total = 25.294 ms GCSServer.deadline_timer.debug_state_event_stats_print - 59 total (1 active, 1 running), Execution time: mean = 1.571 ms, total = 92.698 ms, Queueing time: mean = 65.508 us, max = 182.019 us, min = 15.007 us, total = 3.865 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:00:53,901 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 163378 total (1 active) Queueing time: mean = 48.932 us, max = 7.565 ms, min = 2.216 us, total = 7.994 s Execution time: mean = 252.080 us, total = 41.184 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 74256 total (0 active), Execution time: mean = 521.837 us, total = 38.750 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 74256 total (0 active), Execution time: mean = 27.177 us, total = 2.018 s, Queueing time: mean = 104.898 us, max = 7.565 ms, min = 3.372 us, total = 7.789 s event_loop_lag_probe - 14156 total (0 active), Execution time: mean = 28.865 us, total = 408.608 ms, Queueing time: mean = 10.441 us, max = 4.983 ms, min = 2.216 us, total = 147.804 ms GcsTaskManager.GcJobSummary - 709 total (1 active), Execution time: mean = 11.509 us, total = 8.160 ms, Queueing time: mean = 80.625 us, max = 229.756 us, min = 17.567 us, total = 57.163 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:00:53,901 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17449 total (25 active) Queueing time: mean = 19.432 us, max = 333.684 us, min = 2.330 us, total = 339.066 ms Execution time: mean = 4.187 s, total = 73058.011 s Event stats: event_loop_lag_probe - 14156 total (0 active), Execution time: mean = 31.420 us, total = 444.785 ms, Queueing time: mean = 11.550 us, max = 291.078 us, min = 2.330 us, total = 163.497 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 878 total (0 active), Execution time: mean = 43.429 us, total = 38.131 ms, Queueing time: mean = 103.625 us, max = 333.684 us, min = 8.944 us, total = 90.983 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 878 total (24 active), Execution time: mean = 83.208 s, total = 73056.943 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 737 total (0 active), Execution time: mean = 116.209 us, total = 85.646 ms, Queueing time: mean = 108.879 us, max = 247.252 us, min = 16.251 us, total = 80.244 ms InternalPubSubGcsService.grpc_server.GcsPublish - 737 total (0 active), Execution time: mean = 645.720 us, total = 475.896 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 12 total (1 active), Execution time: mean = 603.938 us, total = 7.247 ms, Queueing time: mean = 77.051 us, max = 153.556 us, min = 48.361 us, total = 924.607 us PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:00:53,901 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14183 total (0 active) Queueing time: mean = 10.075 us, max = 1.423 ms, min = 568.000 ns, total = 142.888 ms Execution time: mean = 27.008 us, total = 383.055 ms Event stats: event_loop_lag_probe - 14157 total (0 active), Execution time: mean = 26.948 us, total = 381.506 ms, Queueing time: mean = 9.960 us, max = 1.423 ms, min = 1.850 us, total = 141.009 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:01:53,901 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 718 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 749 - cumulative published bytes: 592477 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:01:53,902 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 43542 total (4 active) Queueing time: mean = 99.559 us, max = 279.303 ms, min = -0.000 s, total = 4.335 s Execution time: mean = 246.810 us, total = 10.747 s Event stats: event_loop_lag_probe - 14396 total (0 active), Execution time: mean = 24.381 us, total = 350.989 ms, Queueing time: mean = 10.210 us, max = 13.097 ms, min = 2.088 us, total = 146.976 ms GcsInMemoryStore.Put - 5802 total (0 active), Execution time: mean = 53.590 us, total = 310.928 ms, Queueing time: mean = 382.699 us, max = 278.545 ms, min = 2.025 us, total = 2.220 s RayletLoadPulled - 3599 total (1 active), Execution time: mean = 248.723 us, total = 895.153 ms, Queueing time: mean = 74.975 us, max = 472.090 us, min = -0.000 s, total = 269.836 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3596 total (0 active), Execution time: mean = 77.294 us, total = 277.948 ms, Queueing time: mean = 104.925 us, max = 769.233 us, min = 11.910 us, total = 377.310 ms NodeManagerService.grpc_client.GetResourceLoad - 3596 total (0 active), Execution time: mean = 1.490 ms, total = 5.359 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1450 total (0 active), Execution time: mean = 42.366 us, total = 61.430 ms, Queueing time: mean = 103.850 us, max = 228.583 us, min = 11.443 us, total = 150.583 ms InternalKVGcsService.grpc_server.InternalKVPut - 1450 total (0 active), Execution time: mean = 564.002 us, total = 817.803 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1440 total (0 active), Execution time: mean = 34.936 us, total = 50.308 ms, Queueing time: mean = 117.794 us, max = 1.498 ms, min = 20.329 us, total = 169.624 ms NodeInfoGcsService.grpc_server.CheckAlive - 1440 total (0 active), Execution time: mean = 589.820 us, total = 849.341 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1201 total (1 active), Execution time: mean = 9.579 us, total = 11.505 ms, Queueing time: mean = 75.555 us, max = 328.578 us, min = 16.354 us, total = 90.741 ms HealthCheck - 1198 total (0 active), Execution time: mean = 7.674 us, total = 9.194 ms, Queueing time: mean = 102.415 us, max = 399.856 us, min = 16.806 us, total = 122.693 ms GcsInMemoryStore.Get - 787 total (0 active), Execution time: mean = 21.724 us, total = 17.097 ms, Queueing time: mean = 13.354 us, max = 64.163 us, min = 3.701 us, total = 10.509 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 786 total (0 active), Execution time: mean = 55.491 us, total = 43.616 ms, Queueing time: mean = 105.861 us, max = 269.544 us, min = 13.461 us, total = 83.206 ms InternalKVGcsService.grpc_server.InternalKVGet - 786 total (0 active), Execution time: mean = 597.033 us, total = 469.268 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 718 total (0 active), Execution time: mean = 92.301 us, total = 66.272 ms, Queueing time: mean = 117.161 us, max = 296.469 us, min = 19.828 us, total = 84.121 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 718 total (0 active), Execution time: mean = 685.110 us, total = 491.909 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 360 total (1 active), Execution time: mean = 1.421 ms, total = 511.684 ms, Queueing time: mean = 71.486 us, max = 315.299 us, min = -0.000 s, total = 25.735 ms GCSServer.deadline_timer.debug_state_event_stats_print - 60 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 93.533 ms, Queueing time: mean = 64.631 us, max = 182.019 us, min = 12.907 us, total = 3.878 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:01:53,902 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 166150 total (1 active) Queueing time: mean = 48.946 us, max = 7.565 ms, min = 2.216 us, total = 8.132 s Execution time: mean = 252.256 us, total = 41.912 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 75516 total (0 active), Execution time: mean = 522.196 us, total = 39.434 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 75516 total (0 active), Execution time: mean = 27.199 us, total = 2.054 s, Queueing time: mean = 104.925 us, max = 7.565 ms, min = 3.372 us, total = 7.923 s event_loop_lag_probe - 14396 total (0 active), Execution time: mean = 28.888 us, total = 415.878 ms, Queueing time: mean = 10.448 us, max = 4.983 ms, min = 2.216 us, total = 150.415 ms GcsTaskManager.GcJobSummary - 721 total (1 active), Execution time: mean = 11.506 us, total = 8.296 ms, Queueing time: mean = 80.910 us, max = 229.756 us, min = 17.567 us, total = 58.336 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:01:53,902 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 17780 total (25 active) Queueing time: mean = 19.531 us, max = 333.684 us, min = 2.330 us, total = 347.261 ms Execution time: mean = 4.821 s, total = 85718.196 s Event stats: event_loop_lag_probe - 14396 total (0 active), Execution time: mean = 31.447 us, total = 452.707 ms, Queueing time: mean = 11.557 us, max = 291.078 us, min = 2.330 us, total = 166.374 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 911 total (0 active), Execution time: mean = 42.854 us, total = 39.040 ms, Queueing time: mean = 104.119 us, max = 333.684 us, min = 8.944 us, total = 94.852 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 911 total (24 active), Execution time: mean = 94.091 s, total = 85717.108 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 749 total (0 active), Execution time: mean = 116.179 us, total = 87.018 ms, Queueing time: mean = 108.949 us, max = 247.252 us, min = 16.251 us, total = 81.603 ms InternalPubSubGcsService.grpc_server.GcsPublish - 749 total (0 active), Execution time: mean = 646.170 us, total = 483.981 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 665.707 us, total = 8.654 ms, Queueing time: mean = 77.996 us, max = 153.556 us, min = 48.361 us, total = 1.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:01:53,903 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14423 total (0 active) Queueing time: mean = 10.074 us, max = 1.423 ms, min = 568.000 ns, total = 145.296 ms Execution time: mean = 27.015 us, total = 389.644 ms Event stats: event_loop_lag_probe - 14397 total (0 active), Execution time: mean = 26.957 us, total = 388.095 ms, Queueing time: mean = 9.962 us, max = 1.423 ms, min = 1.850 us, total = 143.416 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:02:53,903 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 730 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 761 - cumulative published bytes: 602305 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:02:53,904 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 44261 total (4 active) Queueing time: mean = 98.596 us, max = 279.303 ms, min = -0.000 s, total = 4.364 s Execution time: mean = 246.659 us, total = 10.917 s Event stats: event_loop_lag_probe - 14636 total (0 active), Execution time: mean = 24.440 us, total = 357.703 ms, Queueing time: mean = 10.218 us, max = 13.097 ms, min = 2.088 us, total = 149.553 ms GcsInMemoryStore.Put - 5898 total (0 active), Execution time: mean = 52.806 us, total = 311.450 ms, Queueing time: mean = 377.257 us, max = 278.545 ms, min = 2.025 us, total = 2.225 s RayletLoadPulled - 3659 total (1 active), Execution time: mean = 248.588 us, total = 909.584 ms, Queueing time: mean = 74.946 us, max = 472.090 us, min = -0.000 s, total = 274.229 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3656 total (0 active), Execution time: mean = 77.283 us, total = 282.545 ms, Queueing time: mean = 104.688 us, max = 769.233 us, min = 11.910 us, total = 382.739 ms NodeManagerService.grpc_client.GetResourceLoad - 3656 total (0 active), Execution time: mean = 1.489 ms, total = 5.445 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1474 total (0 active), Execution time: mean = 42.289 us, total = 62.334 ms, Queueing time: mean = 103.796 us, max = 228.583 us, min = 11.443 us, total = 152.995 ms InternalKVGcsService.grpc_server.InternalKVPut - 1474 total (0 active), Execution time: mean = 564.193 us, total = 831.620 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1464 total (0 active), Execution time: mean = 34.908 us, total = 51.106 ms, Queueing time: mean = 117.811 us, max = 1.498 ms, min = 20.329 us, total = 172.475 ms NodeInfoGcsService.grpc_server.CheckAlive - 1464 total (0 active), Execution time: mean = 590.202 us, total = 864.056 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1221 total (1 active), Execution time: mean = 9.604 us, total = 11.726 ms, Queueing time: mean = 75.556 us, max = 328.578 us, min = 16.354 us, total = 92.254 ms HealthCheck - 1218 total (0 active), Execution time: mean = 7.675 us, total = 9.349 ms, Queueing time: mean = 102.280 us, max = 399.856 us, min = 16.806 us, total = 124.577 ms GcsInMemoryStore.Get - 799 total (0 active), Execution time: mean = 21.702 us, total = 17.340 ms, Queueing time: mean = 13.348 us, max = 64.163 us, min = 3.701 us, total = 10.665 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 798 total (0 active), Execution time: mean = 55.550 us, total = 44.329 ms, Queueing time: mean = 105.710 us, max = 269.544 us, min = 13.461 us, total = 84.357 ms InternalKVGcsService.grpc_server.InternalKVGet - 798 total (0 active), Execution time: mean = 597.103 us, total = 476.488 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 730 total (0 active), Execution time: mean = 92.274 us, total = 67.360 ms, Queueing time: mean = 117.128 us, max = 296.469 us, min = 19.828 us, total = 85.503 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 730 total (0 active), Execution time: mean = 685.227 us, total = 500.216 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 366 total (1 active), Execution time: mean = 1.421 ms, total = 520.194 ms, Queueing time: mean = 71.509 us, max = 315.299 us, min = -0.000 s, total = 26.172 ms GCSServer.deadline_timer.debug_state_event_stats_print - 61 total (1 active, 1 running), Execution time: mean = 1.561 ms, total = 95.195 ms, Queueing time: mean = 65.682 us, max = 182.019 us, min = 12.907 us, total = 4.007 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:02:53,904 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 168922 total (1 active) Queueing time: mean = 48.968 us, max = 7.565 ms, min = 2.216 us, total = 8.272 s Execution time: mean = 252.342 us, total = 42.626 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 76776 total (0 active), Execution time: mean = 522.378 us, total = 40.106 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 76776 total (0 active), Execution time: mean = 27.206 us, total = 2.089 s, Queueing time: mean = 104.975 us, max = 7.565 ms, min = 3.372 us, total = 8.060 s event_loop_lag_probe - 14636 total (0 active), Execution time: mean = 28.889 us, total = 422.824 ms, Queueing time: mean = 10.448 us, max = 4.983 ms, min = 2.216 us, total = 152.918 ms GcsTaskManager.GcJobSummary - 733 total (1 active), Execution time: mean = 11.478 us, total = 8.413 ms, Queueing time: mean = 80.768 us, max = 229.756 us, min = 17.567 us, total = 59.203 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:02:53,904 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18068 total (25 active) Queueing time: mean = 19.519 us, max = 333.684 us, min = 2.330 us, total = 352.665 ms Execution time: mean = 4.748 s, total = 85778.390 s Event stats: event_loop_lag_probe - 14636 total (0 active), Execution time: mean = 31.443 us, total = 460.198 ms, Queueing time: mean = 11.555 us, max = 291.078 us, min = 2.330 us, total = 169.124 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 923 total (0 active), Execution time: mean = 42.878 us, total = 39.577 ms, Queueing time: mean = 104.378 us, max = 333.684 us, min = 8.944 us, total = 96.341 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 923 total (24 active), Execution time: mean = 92.933 s, total = 85777.284 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 761 total (0 active), Execution time: mean = 116.228 us, total = 88.450 ms, Queueing time: mean = 108.762 us, max = 247.252 us, min = 16.251 us, total = 82.768 ms InternalPubSubGcsService.grpc_server.GcsPublish - 761 total (0 active), Execution time: mean = 646.435 us, total = 491.937 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 665.707 us, total = 8.654 ms, Queueing time: mean = 77.996 us, max = 153.556 us, min = 48.361 us, total = 1.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:02:53,904 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14663 total (0 active) Queueing time: mean = 10.082 us, max = 1.423 ms, min = 568.000 ns, total = 147.838 ms Execution time: mean = 27.037 us, total = 396.443 ms Event stats: event_loop_lag_probe - 14637 total (0 active), Execution time: mean = 26.979 us, total = 394.894 ms, Queueing time: mean = 9.972 us, max = 1.423 ms, min = 1.850 us, total = 145.959 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:03:53,904 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 742 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 773 - cumulative published bytes: 612133 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:03:53,906 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 44978 total (4 active) Queueing time: mean = 97.689 us, max = 279.303 ms, min = -0.000 s, total = 4.394 s Execution time: mean = 246.729 us, total = 11.097 s Event stats: event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 24.464 us, total = 363.925 ms, Queueing time: mean = 10.221 us, max = 13.097 ms, min = 2.088 us, total = 152.044 ms GcsInMemoryStore.Put - 5994 total (0 active), Execution time: mean = 52.045 us, total = 311.958 ms, Queueing time: mean = 371.943 us, max = 278.545 ms, min = 2.025 us, total = 2.229 s RayletLoadPulled - 3719 total (1 active), Execution time: mean = 248.658 us, total = 924.759 ms, Queueing time: mean = 74.783 us, max = 472.090 us, min = -0.000 s, total = 278.118 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3716 total (0 active), Execution time: mean = 77.252 us, total = 287.068 ms, Queueing time: mean = 104.578 us, max = 769.233 us, min = 11.910 us, total = 388.611 ms NodeManagerService.grpc_client.GetResourceLoad - 3716 total (0 active), Execution time: mean = 1.490 ms, total = 5.538 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1498 total (0 active), Execution time: mean = 42.391 us, total = 63.501 ms, Queueing time: mean = 104.052 us, max = 228.583 us, min = 11.443 us, total = 155.870 ms InternalKVGcsService.grpc_server.InternalKVPut - 1498 total (0 active), Execution time: mean = 565.002 us, total = 846.374 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1487 total (0 active), Execution time: mean = 34.963 us, total = 51.990 ms, Queueing time: mean = 117.766 us, max = 1.498 ms, min = 20.329 us, total = 175.118 ms NodeInfoGcsService.grpc_server.CheckAlive - 1487 total (0 active), Execution time: mean = 590.846 us, total = 878.588 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1241 total (1 active), Execution time: mean = 9.602 us, total = 11.916 ms, Queueing time: mean = 75.835 us, max = 328.578 us, min = 16.354 us, total = 94.111 ms HealthCheck - 1238 total (0 active), Execution time: mean = 7.683 us, total = 9.512 ms, Queueing time: mean = 102.456 us, max = 399.856 us, min = 16.806 us, total = 126.841 ms GcsInMemoryStore.Get - 811 total (0 active), Execution time: mean = 21.700 us, total = 17.598 ms, Queueing time: mean = 13.353 us, max = 64.163 us, min = 3.701 us, total = 10.829 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 810 total (0 active), Execution time: mean = 55.727 us, total = 45.139 ms, Queueing time: mean = 105.776 us, max = 269.544 us, min = 13.461 us, total = 85.679 ms InternalKVGcsService.grpc_server.InternalKVGet - 810 total (0 active), Execution time: mean = 597.594 us, total = 484.051 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 742 total (0 active), Execution time: mean = 92.230 us, total = 68.435 ms, Queueing time: mean = 117.257 us, max = 296.469 us, min = 19.828 us, total = 87.005 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 742 total (0 active), Execution time: mean = 685.266 us, total = 508.467 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 372 total (1 active), Execution time: mean = 1.421 ms, total = 528.794 ms, Queueing time: mean = 71.913 us, max = 315.299 us, min = -0.000 s, total = 26.752 ms GCSServer.deadline_timer.debug_state_event_stats_print - 62 total (1 active, 1 running), Execution time: mean = 1.562 ms, total = 96.870 ms, Queueing time: mean = 65.663 us, max = 182.019 us, min = 12.907 us, total = 4.071 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:03:53,906 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 171694 total (1 active) Queueing time: mean = 49.006 us, max = 7.565 ms, min = 2.216 us, total = 8.414 s Execution time: mean = 252.578 us, total = 43.366 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 78036 total (0 active), Execution time: mean = 522.825 us, total = 40.799 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 78036 total (0 active), Execution time: mean = 27.273 us, total = 2.128 s, Queueing time: mean = 105.058 us, max = 7.565 ms, min = 3.372 us, total = 8.198 s event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 28.910 us, total = 430.063 ms, Queueing time: mean = 10.450 us, max = 4.983 ms, min = 2.216 us, total = 155.448 ms GcsTaskManager.GcJobSummary - 745 total (1 active), Execution time: mean = 11.543 us, total = 8.599 ms, Queueing time: mean = 80.883 us, max = 229.756 us, min = 17.567 us, total = 60.258 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:03:53,906 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18356 total (25 active) Queueing time: mean = 19.533 us, max = 333.684 us, min = 2.330 us, total = 358.553 ms Execution time: mean = 4.676 s, total = 85838.502 s Event stats: event_loop_lag_probe - 14876 total (0 active), Execution time: mean = 31.493 us, total = 468.493 ms, Queueing time: mean = 11.572 us, max = 291.078 us, min = 2.330 us, total = 172.150 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 935 total (0 active), Execution time: mean = 42.969 us, total = 40.176 ms, Queueing time: mean = 104.561 us, max = 333.684 us, min = 8.944 us, total = 97.765 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 935 total (24 active), Execution time: mean = 91.805 s, total = 85837.378 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 773 total (0 active), Execution time: mean = 116.158 us, total = 89.790 ms, Queueing time: mean = 108.936 us, max = 247.252 us, min = 16.251 us, total = 84.207 ms InternalPubSubGcsService.grpc_server.GcsPublish - 773 total (0 active), Execution time: mean = 647.133 us, total = 500.234 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 665.707 us, total = 8.654 ms, Queueing time: mean = 77.996 us, max = 153.556 us, min = 48.361 us, total = 1.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:03:53,906 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 14903 total (0 active) Queueing time: mean = 10.081 us, max = 1.423 ms, min = 568.000 ns, total = 150.240 ms Execution time: mean = 27.038 us, total = 402.949 ms Event stats: event_loop_lag_probe - 14877 total (0 active), Execution time: mean = 26.981 us, total = 401.400 ms, Queueing time: mean = 9.972 us, max = 1.423 ms, min = 1.850 us, total = 148.360 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:04:53,906 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 754 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 785 - cumulative published bytes: 621961 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:04:53,907 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 45697 total (4 active) Queueing time: mean = 96.875 us, max = 279.303 ms, min = -0.000 s, total = 4.427 s Execution time: mean = 246.780 us, total = 11.277 s Event stats: event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 24.524 us, total = 370.697 ms, Queueing time: mean = 10.226 us, max = 13.097 ms, min = 2.088 us, total = 154.580 ms GcsInMemoryStore.Put - 6090 total (0 active), Execution time: mean = 51.310 us, total = 312.477 ms, Queueing time: mean = 366.835 us, max = 278.545 ms, min = 2.025 us, total = 2.234 s RayletLoadPulled - 3779 total (1 active), Execution time: mean = 248.756 us, total = 940.049 ms, Queueing time: mean = 75.003 us, max = 472.090 us, min = -0.000 s, total = 283.437 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3776 total (0 active), Execution time: mean = 77.277 us, total = 291.796 ms, Queueing time: mean = 104.744 us, max = 769.233 us, min = 11.910 us, total = 395.512 ms NodeManagerService.grpc_client.GetResourceLoad - 3776 total (0 active), Execution time: mean = 1.491 ms, total = 5.631 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1522 total (0 active), Execution time: mean = 42.392 us, total = 64.520 ms, Queueing time: mean = 104.170 us, max = 228.583 us, min = 11.443 us, total = 158.547 ms InternalKVGcsService.grpc_server.InternalKVPut - 1522 total (0 active), Execution time: mean = 565.351 us, total = 860.464 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1511 total (0 active), Execution time: mean = 35.015 us, total = 52.908 ms, Queueing time: mean = 117.969 us, max = 1.498 ms, min = 20.329 us, total = 178.251 ms NodeInfoGcsService.grpc_server.CheckAlive - 1511 total (0 active), Execution time: mean = 591.502 us, total = 893.760 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1261 total (1 active), Execution time: mean = 9.603 us, total = 12.109 ms, Queueing time: mean = 75.965 us, max = 328.578 us, min = 16.354 us, total = 95.792 ms HealthCheck - 1258 total (0 active), Execution time: mean = 7.705 us, total = 9.694 ms, Queueing time: mean = 102.710 us, max = 399.856 us, min = 16.806 us, total = 129.209 ms GcsInMemoryStore.Get - 823 total (0 active), Execution time: mean = 21.683 us, total = 17.845 ms, Queueing time: mean = 13.373 us, max = 64.163 us, min = 3.701 us, total = 11.006 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 822 total (0 active), Execution time: mean = 55.786 us, total = 45.856 ms, Queueing time: mean = 105.906 us, max = 269.544 us, min = 13.461 us, total = 87.055 ms InternalKVGcsService.grpc_server.InternalKVGet - 822 total (0 active), Execution time: mean = 598.193 us, total = 491.715 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 754 total (0 active), Execution time: mean = 92.101 us, total = 69.444 ms, Queueing time: mean = 117.645 us, max = 296.469 us, min = 19.828 us, total = 88.704 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 754 total (0 active), Execution time: mean = 685.011 us, total = 516.498 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 378 total (1 active), Execution time: mean = 1.423 ms, total = 537.899 ms, Queueing time: mean = 72.208 us, max = 315.299 us, min = -0.000 s, total = 27.295 ms GCSServer.deadline_timer.debug_state_event_stats_print - 63 total (1 active, 1 running), Execution time: mean = 1.565 ms, total = 98.600 ms, Queueing time: mean = 65.399 us, max = 182.019 us, min = 12.907 us, total = 4.120 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:04:53,908 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 174466 total (1 active) Queueing time: mean = 49.030 us, max = 7.565 ms, min = 2.216 us, total = 8.554 s Execution time: mean = 252.717 us, total = 44.091 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 79296 total (0 active), Execution time: mean = 523.083 us, total = 41.478 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 79296 total (0 active), Execution time: mean = 27.305 us, total = 2.165 s, Queueing time: mean = 105.104 us, max = 7.565 ms, min = 3.372 us, total = 8.334 s event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 28.989 us, total = 438.194 ms, Queueing time: mean = 10.474 us, max = 4.983 ms, min = 2.216 us, total = 158.318 ms GcsTaskManager.GcJobSummary - 757 total (1 active), Execution time: mean = 11.573 us, total = 8.761 ms, Queueing time: mean = 80.931 us, max = 229.756 us, min = 17.567 us, total = 61.265 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:04:53,908 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18644 total (25 active) Queueing time: mean = 19.530 us, max = 333.684 us, min = 2.330 us, total = 364.111 ms Execution time: mean = 4.607 s, total = 85898.734 s Event stats: event_loop_lag_probe - 15116 total (0 active), Execution time: mean = 31.540 us, total = 476.757 ms, Queueing time: mean = 11.584 us, max = 291.078 us, min = 2.330 us, total = 175.098 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 947 total (0 active), Execution time: mean = 43.163 us, total = 40.875 ms, Queueing time: mean = 104.599 us, max = 333.684 us, min = 8.944 us, total = 99.056 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 947 total (24 active), Execution time: mean = 90.705 s, total = 85897.591 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 785 total (0 active), Execution time: mean = 116.248 us, total = 91.255 ms, Queueing time: mean = 108.951 us, max = 247.252 us, min = 16.251 us, total = 85.526 ms InternalPubSubGcsService.grpc_server.GcsPublish - 785 total (0 active), Execution time: mean = 647.613 us, total = 508.376 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 665.707 us, total = 8.654 ms, Queueing time: mean = 77.996 us, max = 153.556 us, min = 48.361 us, total = 1.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:04:53,908 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15143 total (0 active) Queueing time: mean = 10.097 us, max = 1.423 ms, min = 568.000 ns, total = 152.894 ms Execution time: mean = 27.070 us, total = 409.915 ms Event stats: event_loop_lag_probe - 15117 total (0 active), Execution time: mean = 27.014 us, total = 408.366 ms, Queueing time: mean = 9.990 us, max = 1.423 ms, min = 1.850 us, total = 151.015 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:05:53,908 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 766 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 797 - cumulative published bytes: 631789 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:05:53,909 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 46416 total (4 active) Queueing time: mean = 96.050 us, max = 279.303 ms, min = -0.000 s, total = 4.458 s Execution time: mean = 246.941 us, total = 11.462 s Event stats: event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 24.550 us, total = 376.991 ms, Queueing time: mean = 10.216 us, max = 13.097 ms, min = 2.088 us, total = 156.881 ms GcsInMemoryStore.Put - 6186 total (0 active), Execution time: mean = 50.600 us, total = 313.009 ms, Queueing time: mean = 361.926 us, max = 278.545 ms, min = 2.025 us, total = 2.239 s RayletLoadPulled - 3839 total (1 active), Execution time: mean = 248.856 us, total = 955.358 ms, Queueing time: mean = 74.939 us, max = 472.090 us, min = -0.000 s, total = 287.690 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3836 total (0 active), Execution time: mean = 77.353 us, total = 296.727 ms, Queueing time: mean = 104.868 us, max = 769.233 us, min = 11.910 us, total = 402.273 ms NodeManagerService.grpc_client.GetResourceLoad - 3836 total (0 active), Execution time: mean = 1.492 ms, total = 5.724 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1546 total (0 active), Execution time: mean = 42.461 us, total = 65.645 ms, Queueing time: mean = 104.383 us, max = 228.583 us, min = 11.443 us, total = 161.376 ms InternalKVGcsService.grpc_server.InternalKVPut - 1546 total (0 active), Execution time: mean = 565.973 us, total = 874.994 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1535 total (0 active), Execution time: mean = 34.993 us, total = 53.714 ms, Queueing time: mean = 117.756 us, max = 1.498 ms, min = 20.329 us, total = 180.755 ms NodeInfoGcsService.grpc_server.CheckAlive - 1535 total (0 active), Execution time: mean = 591.644 us, total = 908.173 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1281 total (1 active), Execution time: mean = 9.621 us, total = 12.325 ms, Queueing time: mean = 76.074 us, max = 328.578 us, min = 16.354 us, total = 97.450 ms HealthCheck - 1278 total (0 active), Execution time: mean = 7.722 us, total = 9.868 ms, Queueing time: mean = 102.933 us, max = 399.856 us, min = 16.806 us, total = 131.549 ms GcsInMemoryStore.Get - 835 total (0 active), Execution time: mean = 21.678 us, total = 18.101 ms, Queueing time: mean = 13.374 us, max = 64.163 us, min = 3.701 us, total = 11.168 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 834 total (0 active), Execution time: mean = 55.859 us, total = 46.586 ms, Queueing time: mean = 106.192 us, max = 269.544 us, min = 13.461 us, total = 88.564 ms InternalKVGcsService.grpc_server.InternalKVGet - 834 total (0 active), Execution time: mean = 605.047 us, total = 504.609 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 766 total (0 active), Execution time: mean = 92.222 us, total = 70.642 ms, Queueing time: mean = 118.088 us, max = 296.469 us, min = 19.828 us, total = 90.456 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 766 total (0 active), Execution time: mean = 685.437 us, total = 525.045 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 384 total (1 active), Execution time: mean = 1.421 ms, total = 545.822 ms, Queueing time: mean = 72.041 us, max = 315.299 us, min = -0.000 s, total = 27.664 ms GCSServer.deadline_timer.debug_state_event_stats_print - 64 total (1 active, 1 running), Execution time: mean = 1.569 ms, total = 100.420 ms, Queueing time: mean = 65.381 us, max = 182.019 us, min = 12.907 us, total = 4.184 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:05:53,909 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 177237 total (1 active) Queueing time: mean = 49.092 us, max = 7.565 ms, min = 2.216 us, total = 8.701 s Execution time: mean = 252.946 us, total = 44.831 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 80556 total (0 active), Execution time: mean = 523.535 us, total = 42.174 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 80556 total (0 active), Execution time: mean = 27.343 us, total = 2.203 s, Queueing time: mean = 105.237 us, max = 7.565 ms, min = 3.372 us, total = 8.477 s event_loop_lag_probe - 15355 total (0 active), Execution time: mean = 29.039 us, total = 445.888 ms, Queueing time: mean = 10.487 us, max = 4.983 ms, min = 2.216 us, total = 161.034 ms GcsTaskManager.GcJobSummary - 769 total (1 active), Execution time: mean = 11.622 us, total = 8.937 ms, Queueing time: mean = 81.009 us, max = 229.756 us, min = 17.567 us, total = 62.296 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:05:53,909 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 18931 total (25 active) Queueing time: mean = 19.544 us, max = 333.684 us, min = 2.330 us, total = 369.982 ms Execution time: mean = 4.541 s, total = 85958.955 s Event stats: event_loop_lag_probe - 15355 total (0 active), Execution time: mean = 31.592 us, total = 485.089 ms, Queueing time: mean = 11.601 us, max = 291.078 us, min = 2.330 us, total = 178.132 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 959 total (0 active), Execution time: mean = 43.248 us, total = 41.475 ms, Queueing time: mean = 104.780 us, max = 333.684 us, min = 8.944 us, total = 100.484 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 959 total (24 active), Execution time: mean = 89.633 s, total = 85957.793 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 797 total (0 active), Execution time: mean = 116.425 us, total = 92.791 ms, Queueing time: mean = 109.079 us, max = 247.252 us, min = 16.251 us, total = 86.936 ms InternalPubSubGcsService.grpc_server.GcsPublish - 797 total (0 active), Execution time: mean = 648.528 us, total = 516.877 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 13 total (1 active), Execution time: mean = 665.707 us, total = 8.654 ms, Queueing time: mean = 77.996 us, max = 153.556 us, min = 48.361 us, total = 1.014 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:05:53,910 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15382 total (0 active) Queueing time: mean = 10.095 us, max = 1.423 ms, min = 568.000 ns, total = 155.281 ms Execution time: mean = 27.070 us, total = 416.395 ms Event stats: event_loop_lag_probe - 15356 total (0 active), Execution time: mean = 27.015 us, total = 414.846 ms, Queueing time: mean = 9.990 us, max = 1.423 ms, min = 1.850 us, total = 153.401 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:06:53,910 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 778 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 809 - cumulative published bytes: 641617 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:06:53,911 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 47135 total (4 active) Queueing time: mean = 95.218 us, max = 279.303 ms, min = -0.000 s, total = 4.488 s Execution time: mean = 247.015 us, total = 11.643 s Event stats: event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 24.570 us, total = 383.187 ms, Queueing time: mean = 10.228 us, max = 13.097 ms, min = 1.994 us, total = 159.521 ms GcsInMemoryStore.Put - 6282 total (0 active), Execution time: mean = 49.917 us, total = 313.576 ms, Queueing time: mean = 357.102 us, max = 278.545 ms, min = 2.025 us, total = 2.243 s RayletLoadPulled - 3899 total (1 active), Execution time: mean = 249.030 us, total = 970.966 ms, Queueing time: mean = 75.043 us, max = 472.090 us, min = -0.000 s, total = 292.591 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3896 total (0 active), Execution time: mean = 77.404 us, total = 301.567 ms, Queueing time: mean = 104.895 us, max = 769.233 us, min = 11.910 us, total = 408.671 ms NodeManagerService.grpc_client.GetResourceLoad - 3896 total (0 active), Execution time: mean = 1.493 ms, total = 5.817 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1570 total (0 active), Execution time: mean = 42.458 us, total = 66.659 ms, Queueing time: mean = 104.210 us, max = 228.583 us, min = 11.443 us, total = 163.609 ms InternalKVGcsService.grpc_server.InternalKVPut - 1570 total (0 active), Execution time: mean = 565.735 us, total = 888.203 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1559 total (0 active), Execution time: mean = 34.968 us, total = 54.515 ms, Queueing time: mean = 117.631 us, max = 1.498 ms, min = 20.329 us, total = 183.386 ms NodeInfoGcsService.grpc_server.CheckAlive - 1559 total (0 active), Execution time: mean = 591.429 us, total = 922.039 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1301 total (1 active), Execution time: mean = 9.637 us, total = 12.538 ms, Queueing time: mean = 76.074 us, max = 328.578 us, min = 16.354 us, total = 98.973 ms HealthCheck - 1298 total (0 active), Execution time: mean = 7.719 us, total = 10.019 ms, Queueing time: mean = 102.802 us, max = 399.856 us, min = 16.806 us, total = 133.437 ms GcsInMemoryStore.Get - 847 total (0 active), Execution time: mean = 21.658 us, total = 18.344 ms, Queueing time: mean = 13.382 us, max = 64.163 us, min = 3.701 us, total = 11.335 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 846 total (0 active), Execution time: mean = 56.016 us, total = 47.389 ms, Queueing time: mean = 106.192 us, max = 269.544 us, min = 13.461 us, total = 89.839 ms InternalKVGcsService.grpc_server.InternalKVGet - 846 total (0 active), Execution time: mean = 610.782 us, total = 516.721 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 778 total (0 active), Execution time: mean = 92.549 us, total = 72.003 ms, Queueing time: mean = 117.866 us, max = 296.469 us, min = 19.828 us, total = 91.699 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 778 total (0 active), Execution time: mean = 685.447 us, total = 533.278 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 390 total (1 active), Execution time: mean = 1.419 ms, total = 553.475 ms, Queueing time: mean = 72.018 us, max = 315.299 us, min = -0.000 s, total = 28.087 ms GCSServer.deadline_timer.debug_state_event_stats_print - 65 total (1 active, 1 running), Execution time: mean = 1.569 ms, total = 101.954 ms, Queueing time: mean = 65.765 us, max = 182.019 us, min = 12.907 us, total = 4.275 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:06:53,911 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 180009 total (1 active) Queueing time: mean = 49.102 us, max = 7.565 ms, min = 2.216 us, total = 8.839 s Execution time: mean = 253.003 us, total = 45.543 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 81816 total (0 active), Execution time: mean = 523.624 us, total = 42.841 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 81816 total (0 active), Execution time: mean = 27.376 us, total = 2.240 s, Queueing time: mean = 105.258 us, max = 7.565 ms, min = 3.372 us, total = 8.612 s event_loop_lag_probe - 15595 total (0 active), Execution time: mean = 29.061 us, total = 453.201 ms, Queueing time: mean = 10.499 us, max = 4.983 ms, min = 2.216 us, total = 163.727 ms GcsTaskManager.GcJobSummary - 781 total (1 active), Execution time: mean = 11.670 us, total = 9.115 ms, Queueing time: mean = 80.963 us, max = 229.756 us, min = 17.567 us, total = 63.232 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:06:53,911 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19224 total (25 active) Queueing time: mean = 19.555 us, max = 333.684 us, min = 2.330 us, total = 375.929 ms Execution time: mean = 4.537 s, total = 87219.068 s Event stats: event_loop_lag_probe - 15595 total (0 active), Execution time: mean = 31.609 us, total = 492.940 ms, Queueing time: mean = 11.606 us, max = 291.078 us, min = 2.330 us, total = 180.989 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 973 total (0 active), Execution time: mean = 43.311 us, total = 42.142 ms, Queueing time: mean = 104.837 us, max = 333.684 us, min = 8.944 us, total = 102.006 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 973 total (24 active), Execution time: mean = 89.638 s, total = 87217.888 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 809 total (0 active), Execution time: mean = 116.366 us, total = 94.140 ms, Queueing time: mean = 109.264 us, max = 247.252 us, min = 16.251 us, total = 88.394 ms InternalPubSubGcsService.grpc_server.GcsPublish - 809 total (0 active), Execution time: mean = 648.976 us, total = 525.022 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 623.424 us, total = 8.728 ms, Queueing time: mean = 80.176 us, max = 153.556 us, min = 48.361 us, total = 1.122 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:06:53,911 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15622 total (0 active) Queueing time: mean = 10.091 us, max = 1.423 ms, min = 568.000 ns, total = 157.648 ms Execution time: mean = 27.078 us, total = 423.010 ms Event stats: event_loop_lag_probe - 15596 total (0 active), Execution time: mean = 27.024 us, total = 421.461 ms, Queueing time: mean = 9.988 us, max = 1.423 ms, min = 1.850 us, total = 155.769 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:07:53,911 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 790 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 821 - cumulative published bytes: 651445 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:07:53,912 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 47854 total (4 active) Queueing time: mean = 94.410 us, max = 279.303 ms, min = -0.000 s, total = 4.518 s Execution time: mean = 247.060 us, total = 11.823 s Event stats: event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 24.590 us, total = 389.414 ms, Queueing time: mean = 10.218 us, max = 13.097 ms, min = 1.994 us, total = 161.809 ms GcsInMemoryStore.Put - 6378 total (0 active), Execution time: mean = 49.492 us, total = 315.660 ms, Queueing time: mean = 352.536 us, max = 278.545 ms, min = 2.025 us, total = 2.248 s RayletLoadPulled - 3959 total (1 active), Execution time: mean = 249.209 us, total = 986.617 ms, Queueing time: mean = 75.090 us, max = 472.090 us, min = -0.000 s, total = 297.281 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 3956 total (0 active), Execution time: mean = 77.421 us, total = 306.277 ms, Queueing time: mean = 104.833 us, max = 769.233 us, min = 11.910 us, total = 414.718 ms NodeManagerService.grpc_client.GetResourceLoad - 3956 total (0 active), Execution time: mean = 1.493 ms, total = 5.908 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1594 total (0 active), Execution time: mean = 42.444 us, total = 67.656 ms, Queueing time: mean = 103.985 us, max = 228.583 us, min = 11.443 us, total = 165.752 ms InternalKVGcsService.grpc_server.InternalKVPut - 1594 total (0 active), Execution time: mean = 565.219 us, total = 900.959 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1583 total (0 active), Execution time: mean = 35.099 us, total = 55.561 ms, Queueing time: mean = 117.548 us, max = 1.498 ms, min = 20.329 us, total = 186.078 ms NodeInfoGcsService.grpc_server.CheckAlive - 1583 total (0 active), Execution time: mean = 592.414 us, total = 937.791 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1321 total (1 active), Execution time: mean = 9.640 us, total = 12.734 ms, Queueing time: mean = 76.106 us, max = 328.578 us, min = 16.354 us, total = 100.536 ms HealthCheck - 1318 total (0 active), Execution time: mean = 7.725 us, total = 10.181 ms, Queueing time: mean = 102.748 us, max = 399.856 us, min = 16.806 us, total = 135.422 ms GcsInMemoryStore.Get - 859 total (0 active), Execution time: mean = 21.635 us, total = 18.585 ms, Queueing time: mean = 13.364 us, max = 64.163 us, min = 3.701 us, total = 11.480 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 858 total (0 active), Execution time: mean = 56.149 us, total = 48.176 ms, Queueing time: mean = 106.044 us, max = 269.544 us, min = 13.461 us, total = 90.986 ms InternalKVGcsService.grpc_server.InternalKVGet - 858 total (0 active), Execution time: mean = 610.575 us, total = 523.874 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 790 total (0 active), Execution time: mean = 92.702 us, total = 73.234 ms, Queueing time: mean = 117.852 us, max = 296.469 us, min = 19.828 us, total = 93.103 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 790 total (0 active), Execution time: mean = 685.640 us, total = 541.655 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 396 total (1 active), Execution time: mean = 1.421 ms, total = 562.679 ms, Queueing time: mean = 72.110 us, max = 315.299 us, min = -0.000 s, total = 28.556 ms GCSServer.deadline_timer.debug_state_event_stats_print - 66 total (1 active, 1 running), Execution time: mean = 1.569 ms, total = 103.562 ms, Queueing time: mean = 65.732 us, max = 182.019 us, min = 12.907 us, total = 4.338 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:07:53,913 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 182781 total (1 active) Queueing time: mean = 49.120 us, max = 7.565 ms, min = 2.216 us, total = 8.978 s Execution time: mean = 253.185 us, total = 46.277 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 83076 total (0 active), Execution time: mean = 523.990 us, total = 43.531 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 83076 total (0 active), Execution time: mean = 27.407 us, total = 2.277 s, Queueing time: mean = 105.296 us, max = 7.565 ms, min = 3.372 us, total = 8.748 s event_loop_lag_probe - 15835 total (0 active), Execution time: mean = 29.067 us, total = 460.276 ms, Queueing time: mean = 10.501 us, max = 4.983 ms, min = 2.216 us, total = 166.284 ms GcsTaskManager.GcJobSummary - 793 total (1 active), Execution time: mean = 11.713 us, total = 9.288 ms, Queueing time: mean = 81.013 us, max = 229.756 us, min = 17.567 us, total = 64.244 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:07:53,913 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19512 total (25 active) Queueing time: mean = 19.545 us, max = 333.684 us, min = 2.330 us, total = 381.367 ms Execution time: mean = 4.473 s, total = 87279.281 s Event stats: event_loop_lag_probe - 15835 total (0 active), Execution time: mean = 31.619 us, total = 500.692 ms, Queueing time: mean = 11.615 us, max = 291.078 us, min = 2.330 us, total = 183.921 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 985 total (0 active), Execution time: mean = 43.475 us, total = 42.822 ms, Queueing time: mean = 104.757 us, max = 333.684 us, min = 8.944 us, total = 103.186 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 985 total (24 active), Execution time: mean = 88.607 s, total = 87278.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 821 total (0 active), Execution time: mean = 116.461 us, total = 95.615 ms, Queueing time: mean = 109.282 us, max = 247.252 us, min = 16.251 us, total = 89.721 ms InternalPubSubGcsService.grpc_server.GcsPublish - 821 total (0 active), Execution time: mean = 649.062 us, total = 532.880 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 623.424 us, total = 8.728 ms, Queueing time: mean = 80.176 us, max = 153.556 us, min = 48.361 us, total = 1.122 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:07:53,913 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 15862 total (0 active) Queueing time: mean = 10.093 us, max = 1.423 ms, min = 568.000 ns, total = 160.088 ms Execution time: mean = 27.102 us, total = 429.889 ms Event stats: event_loop_lag_probe - 15836 total (0 active), Execution time: mean = 27.048 us, total = 428.340 ms, Queueing time: mean = 9.990 us, max = 1.423 ms, min = 1.850 us, total = 158.208 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:08:53,913 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 802 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 833 - cumulative published bytes: 661273 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:08:53,914 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 48573 total (4 active) Queueing time: mean = 93.644 us, max = 279.303 ms, min = -0.000 s, total = 4.549 s Execution time: mean = 246.978 us, total = 11.996 s Event stats: event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 24.609 us, total = 395.620 ms, Queueing time: mean = 10.204 us, max = 13.097 ms, min = 1.994 us, total = 164.043 ms GcsInMemoryStore.Put - 6474 total (0 active), Execution time: mean = 48.836 us, total = 316.167 ms, Queueing time: mean = 348.211 us, max = 278.545 ms, min = 2.025 us, total = 2.254 s RayletLoadPulled - 4019 total (1 active), Execution time: mean = 249.081 us, total = 1.001 s, Queueing time: mean = 75.131 us, max = 472.090 us, min = -0.000 s, total = 301.950 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4016 total (0 active), Execution time: mean = 77.393 us, total = 310.812 ms, Queueing time: mean = 104.640 us, max = 769.233 us, min = 11.910 us, total = 420.232 ms NodeManagerService.grpc_client.GetResourceLoad - 4016 total (0 active), Execution time: mean = 1.493 ms, total = 5.996 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1618 total (0 active), Execution time: mean = 42.484 us, total = 68.739 ms, Queueing time: mean = 104.067 us, max = 228.583 us, min = 11.443 us, total = 168.380 ms InternalKVGcsService.grpc_server.InternalKVPut - 1618 total (0 active), Execution time: mean = 565.700 us, total = 915.302 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1607 total (0 active), Execution time: mean = 35.078 us, total = 56.370 ms, Queueing time: mean = 117.583 us, max = 1.498 ms, min = 20.329 us, total = 188.956 ms NodeInfoGcsService.grpc_server.CheckAlive - 1607 total (0 active), Execution time: mean = 592.885 us, total = 952.766 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1341 total (1 active), Execution time: mean = 9.631 us, total = 12.915 ms, Queueing time: mean = 76.030 us, max = 328.578 us, min = 16.354 us, total = 101.957 ms HealthCheck - 1338 total (0 active), Execution time: mean = 7.722 us, total = 10.332 ms, Queueing time: mean = 102.551 us, max = 399.856 us, min = 16.806 us, total = 137.214 ms GcsInMemoryStore.Get - 871 total (0 active), Execution time: mean = 21.602 us, total = 18.815 ms, Queueing time: mean = 13.362 us, max = 64.163 us, min = 3.701 us, total = 11.638 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 870 total (0 active), Execution time: mean = 56.187 us, total = 48.883 ms, Queueing time: mean = 106.212 us, max = 269.544 us, min = 13.461 us, total = 92.405 ms InternalKVGcsService.grpc_server.InternalKVGet - 870 total (0 active), Execution time: mean = 610.678 us, total = 531.290 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 802 total (0 active), Execution time: mean = 92.686 us, total = 74.334 ms, Queueing time: mean = 117.954 us, max = 296.469 us, min = 19.828 us, total = 94.599 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 802 total (0 active), Execution time: mean = 686.199 us, total = 550.332 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 402 total (1 active), Execution time: mean = 1.421 ms, total = 571.186 ms, Queueing time: mean = 72.327 us, max = 315.299 us, min = -0.000 s, total = 29.075 ms GCSServer.deadline_timer.debug_state_event_stats_print - 67 total (1 active, 1 running), Execution time: mean = 1.572 ms, total = 105.296 ms, Queueing time: mean = 65.802 us, max = 182.019 us, min = 12.907 us, total = 4.409 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:08:53,914 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 185553 total (1 active) Queueing time: mean = 49.151 us, max = 7.565 ms, min = 2.216 us, total = 9.120 s Execution time: mean = 253.311 us, total = 47.003 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 84336 total (0 active), Execution time: mean = 524.247 us, total = 44.213 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 84336 total (0 active), Execution time: mean = 27.428 us, total = 2.313 s, Queueing time: mean = 105.366 us, max = 7.565 ms, min = 3.372 us, total = 8.886 s event_loop_lag_probe - 16075 total (0 active), Execution time: mean = 29.051 us, total = 466.996 ms, Queueing time: mean = 10.495 us, max = 4.983 ms, min = 2.216 us, total = 168.702 ms GcsTaskManager.GcJobSummary - 805 total (1 active), Execution time: mean = 11.716 us, total = 9.431 ms, Queueing time: mean = 80.981 us, max = 229.756 us, min = 17.567 us, total = 65.189 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:08:53,915 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 19800 total (25 active) Queueing time: mean = 19.545 us, max = 333.684 us, min = 2.330 us, total = 386.983 ms Execution time: mean = 4.411 s, total = 87339.501 s Event stats: event_loop_lag_probe - 16075 total (0 active), Execution time: mean = 31.644 us, total = 508.677 ms, Queueing time: mean = 11.629 us, max = 291.078 us, min = 2.330 us, total = 186.938 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 997 total (0 active), Execution time: mean = 43.541 us, total = 43.410 ms, Queueing time: mean = 104.689 us, max = 333.684 us, min = 8.944 us, total = 104.375 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 997 total (24 active), Execution time: mean = 87.601 s, total = 87338.285 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 833 total (0 active), Execution time: mean = 116.795 us, total = 97.290 ms, Queueing time: mean = 109.400 us, max = 247.252 us, min = 16.251 us, total = 91.130 ms InternalPubSubGcsService.grpc_server.GcsPublish - 833 total (0 active), Execution time: mean = 649.923 us, total = 541.386 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 623.424 us, total = 8.728 ms, Queueing time: mean = 80.176 us, max = 153.556 us, min = 48.361 us, total = 1.122 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:08:53,915 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16102 total (0 active) Queueing time: mean = 10.088 us, max = 1.423 ms, min = 568.000 ns, total = 162.443 ms Execution time: mean = 27.093 us, total = 436.255 ms Event stats: event_loop_lag_probe - 16076 total (0 active), Execution time: mean = 27.041 us, total = 434.706 ms, Queueing time: mean = 9.988 us, max = 1.423 ms, min = 1.850 us, total = 160.564 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:09:53,915 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 814 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 845 - cumulative published bytes: 671101 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:09:53,916 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 49292 total (4 active) Queueing time: mean = 92.912 us, max = 279.303 ms, min = -0.000 s, total = 4.580 s Execution time: mean = 247.039 us, total = 12.177 s Event stats: event_loop_lag_probe - 16316 total (0 active), Execution time: mean = 24.628 us, total = 401.825 ms, Queueing time: mean = 10.194 us, max = 13.097 ms, min = 1.994 us, total = 166.323 ms GcsInMemoryStore.Put - 6570 total (0 active), Execution time: mean = 48.201 us, total = 316.681 ms, Queueing time: mean = 343.828 us, max = 278.545 ms, min = 2.025 us, total = 2.259 s RayletLoadPulled - 4079 total (1 active), Execution time: mean = 249.070 us, total = 1.016 s, Queueing time: mean = 75.153 us, max = 472.090 us, min = -0.000 s, total = 306.549 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4076 total (0 active), Execution time: mean = 77.376 us, total = 315.384 ms, Queueing time: mean = 104.783 us, max = 769.233 us, min = 11.910 us, total = 427.095 ms NodeManagerService.grpc_client.GetResourceLoad - 4076 total (0 active), Execution time: mean = 1.494 ms, total = 6.090 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1642 total (0 active), Execution time: mean = 42.507 us, total = 69.796 ms, Queueing time: mean = 104.195 us, max = 228.583 us, min = 11.443 us, total = 171.088 ms InternalKVGcsService.grpc_server.InternalKVPut - 1642 total (0 active), Execution time: mean = 566.245 us, total = 929.774 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1631 total (0 active), Execution time: mean = 35.365 us, total = 57.681 ms, Queueing time: mean = 117.604 us, max = 1.498 ms, min = 20.329 us, total = 191.812 ms NodeInfoGcsService.grpc_server.CheckAlive - 1631 total (0 active), Execution time: mean = 593.461 us, total = 967.935 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1361 total (1 active), Execution time: mean = 9.622 us, total = 13.095 ms, Queueing time: mean = 76.088 us, max = 328.578 us, min = 16.354 us, total = 103.555 ms HealthCheck - 1358 total (0 active), Execution time: mean = 7.732 us, total = 10.500 ms, Queueing time: mean = 102.684 us, max = 399.856 us, min = 16.806 us, total = 139.444 ms GcsInMemoryStore.Get - 883 total (0 active), Execution time: mean = 21.582 us, total = 19.057 ms, Queueing time: mean = 13.352 us, max = 64.163 us, min = 3.701 us, total = 11.789 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 882 total (0 active), Execution time: mean = 56.216 us, total = 49.582 ms, Queueing time: mean = 106.298 us, max = 269.544 us, min = 13.461 us, total = 93.755 ms InternalKVGcsService.grpc_server.InternalKVGet - 882 total (0 active), Execution time: mean = 611.124 us, total = 539.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 814 total (0 active), Execution time: mean = 92.649 us, total = 75.416 ms, Queueing time: mean = 118.137 us, max = 296.469 us, min = 19.828 us, total = 96.164 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 814 total (0 active), Execution time: mean = 686.191 us, total = 558.559 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 408 total (1 active), Execution time: mean = 1.422 ms, total = 580.058 ms, Queueing time: mean = 72.179 us, max = 315.299 us, min = -0.000 s, total = 29.449 ms GCSServer.deadline_timer.debug_state_event_stats_print - 68 total (1 active, 1 running), Execution time: mean = 1.575 ms, total = 107.072 ms, Queueing time: mean = 65.770 us, max = 182.019 us, min = 12.907 us, total = 4.472 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:09:53,916 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 188325 total (1 active) Queueing time: mean = 49.215 us, max = 7.565 ms, min = 2.216 us, total = 9.268 s Execution time: mean = 253.490 us, total = 47.739 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 85596 total (0 active), Execution time: mean = 524.575 us, total = 44.902 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 85596 total (0 active), Execution time: mean = 27.496 us, total = 2.354 s, Queueing time: mean = 105.509 us, max = 7.565 ms, min = 3.372 us, total = 9.031 s event_loop_lag_probe - 16315 total (0 active), Execution time: mean = 29.046 us, total = 473.891 ms, Queueing time: mean = 10.487 us, max = 4.983 ms, min = 2.216 us, total = 171.099 ms GcsTaskManager.GcJobSummary - 817 total (1 active), Execution time: mean = 11.741 us, total = 9.592 ms, Queueing time: mean = 80.951 us, max = 229.756 us, min = 17.567 us, total = 66.137 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:09:53,916 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20088 total (25 active) Queueing time: mean = 19.547 us, max = 333.684 us, min = 2.330 us, total = 392.658 ms Execution time: mean = 4.351 s, total = 87399.601 s Event stats: event_loop_lag_probe - 16315 total (0 active), Execution time: mean = 31.670 us, total = 516.694 ms, Queueing time: mean = 11.643 us, max = 291.078 us, min = 2.330 us, total = 189.948 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1009 total (0 active), Execution time: mean = 43.636 us, total = 44.028 ms, Queueing time: mean = 104.725 us, max = 333.684 us, min = 8.944 us, total = 105.668 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1009 total (24 active), Execution time: mean = 86.619 s, total = 87398.367 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 845 total (0 active), Execution time: mean = 116.664 us, total = 98.581 ms, Queueing time: mean = 109.470 us, max = 247.252 us, min = 16.251 us, total = 92.502 ms InternalPubSubGcsService.grpc_server.GcsPublish - 845 total (0 active), Execution time: mean = 649.914 us, total = 549.177 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 623.424 us, total = 8.728 ms, Queueing time: mean = 80.176 us, max = 153.556 us, min = 48.361 us, total = 1.122 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:09:53,917 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16342 total (0 active) Queueing time: mean = 10.088 us, max = 1.423 ms, min = 568.000 ns, total = 164.856 ms Execution time: mean = 27.111 us, total = 443.045 ms Event stats: event_loop_lag_probe - 16316 total (0 active), Execution time: mean = 27.059 us, total = 441.496 ms, Queueing time: mean = 9.989 us, max = 1.423 ms, min = 1.850 us, total = 162.977 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:10:53,917 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 826 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 857 - cumulative published bytes: 680929 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:10:53,917 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 50011 total (4 active) Queueing time: mean = 92.174 us, max = 279.303 ms, min = -0.000 s, total = 4.610 s Execution time: mean = 247.044 us, total = 12.355 s Event stats: event_loop_lag_probe - 16556 total (0 active), Execution time: mean = 24.649 us, total = 408.093 ms, Queueing time: mean = 10.180 us, max = 13.097 ms, min = 1.994 us, total = 168.544 ms GcsInMemoryStore.Put - 6666 total (0 active), Execution time: mean = 47.581 us, total = 317.172 ms, Queueing time: mean = 339.452 us, max = 278.545 ms, min = 2.025 us, total = 2.263 s RayletLoadPulled - 4139 total (1 active), Execution time: mean = 249.035 us, total = 1.031 s, Queueing time: mean = 75.275 us, max = 472.090 us, min = -0.000 s, total = 311.563 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4136 total (0 active), Execution time: mean = 77.415 us, total = 320.188 ms, Queueing time: mean = 104.822 us, max = 769.233 us, min = 11.910 us, total = 433.545 ms NodeManagerService.grpc_client.GetResourceLoad - 4136 total (0 active), Execution time: mean = 1.494 ms, total = 6.181 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1666 total (0 active), Execution time: mean = 42.559 us, total = 70.903 ms, Queueing time: mean = 104.208 us, max = 228.583 us, min = 11.443 us, total = 173.611 ms InternalKVGcsService.grpc_server.InternalKVPut - 1666 total (0 active), Execution time: mean = 566.943 us, total = 944.527 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1655 total (0 active), Execution time: mean = 35.442 us, total = 58.656 ms, Queueing time: mean = 117.535 us, max = 1.498 ms, min = 20.329 us, total = 194.521 ms NodeInfoGcsService.grpc_server.CheckAlive - 1655 total (0 active), Execution time: mean = 593.792 us, total = 982.726 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1381 total (1 active), Execution time: mean = 9.606 us, total = 13.266 ms, Queueing time: mean = 75.881 us, max = 328.578 us, min = 16.354 us, total = 104.792 ms HealthCheck - 1378 total (0 active), Execution time: mean = 7.737 us, total = 10.661 ms, Queueing time: mean = 102.910 us, max = 399.856 us, min = 16.806 us, total = 141.810 ms GcsInMemoryStore.Get - 895 total (0 active), Execution time: mean = 21.581 us, total = 19.315 ms, Queueing time: mean = 13.343 us, max = 64.163 us, min = 3.701 us, total = 11.942 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 894 total (0 active), Execution time: mean = 56.197 us, total = 50.240 ms, Queueing time: mean = 106.469 us, max = 269.544 us, min = 13.461 us, total = 95.183 ms InternalKVGcsService.grpc_server.InternalKVGet - 894 total (0 active), Execution time: mean = 611.420 us, total = 546.609 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 826 total (0 active), Execution time: mean = 92.695 us, total = 76.566 ms, Queueing time: mean = 118.260 us, max = 296.469 us, min = 19.828 us, total = 97.683 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 826 total (0 active), Execution time: mean = 686.630 us, total = 567.156 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 414 total (1 active), Execution time: mean = 1.421 ms, total = 588.337 ms, Queueing time: mean = 72.049 us, max = 315.299 us, min = -0.000 s, total = 29.828 ms GCSServer.deadline_timer.debug_state_event_stats_print - 69 total (1 active, 1 running), Execution time: mean = 1.576 ms, total = 108.715 ms, Queueing time: mean = 65.517 us, max = 182.019 us, min = 12.907 us, total = 4.521 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:10:53,917 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 191097 total (1 active) Queueing time: mean = 49.254 us, max = 7.565 ms, min = 2.216 us, total = 9.412 s Execution time: mean = 253.637 us, total = 48.469 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 86856 total (0 active), Execution time: mean = 524.863 us, total = 45.587 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 86856 total (0 active), Execution time: mean = 27.531 us, total = 2.391 s, Queueing time: mean = 105.593 us, max = 7.565 ms, min = 3.372 us, total = 9.171 s event_loop_lag_probe - 16555 total (0 active), Execution time: mean = 29.039 us, total = 480.733 ms, Queueing time: mean = 10.487 us, max = 4.983 ms, min = 2.216 us, total = 173.610 ms GcsTaskManager.GcJobSummary - 829 total (1 active), Execution time: mean = 11.716 us, total = 9.713 ms, Queueing time: mean = 80.982 us, max = 229.756 us, min = 17.567 us, total = 67.134 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:10:53,917 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20376 total (25 active) Queueing time: mean = 19.540 us, max = 333.684 us, min = 2.330 us, total = 398.142 ms Execution time: mean = 4.292 s, total = 87459.822 s Event stats: event_loop_lag_probe - 16555 total (0 active), Execution time: mean = 31.700 us, total = 524.791 ms, Queueing time: mean = 11.658 us, max = 291.078 us, min = 2.330 us, total = 192.996 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1021 total (0 active), Execution time: mean = 43.667 us, total = 44.584 ms, Queueing time: mean = 104.654 us, max = 333.684 us, min = 8.944 us, total = 106.852 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1021 total (24 active), Execution time: mean = 85.660 s, total = 87458.570 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 857 total (0 active), Execution time: mean = 116.831 us, total = 100.125 ms, Queueing time: mean = 109.399 us, max = 247.252 us, min = 16.251 us, total = 93.755 ms InternalPubSubGcsService.grpc_server.GcsPublish - 857 total (0 active), Execution time: mean = 650.369 us, total = 557.366 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 14 total (1 active), Execution time: mean = 623.424 us, total = 8.728 ms, Queueing time: mean = 80.176 us, max = 153.556 us, min = 48.361 us, total = 1.122 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:10:53,917 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16582 total (0 active) Queueing time: mean = 10.092 us, max = 1.423 ms, min = 568.000 ns, total = 167.351 ms Execution time: mean = 27.114 us, total = 449.610 ms Event stats: event_loop_lag_probe - 16556 total (0 active), Execution time: mean = 27.063 us, total = 448.061 ms, Queueing time: mean = 9.995 us, max = 1.423 ms, min = 1.850 us, total = 165.472 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:11:53,918 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 838 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 869 - cumulative published bytes: 690757 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:11:53,919 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 50729 total (4 active) Queueing time: mean = 91.460 us, max = 279.303 ms, min = -0.000 s, total = 4.640 s Execution time: mean = 246.932 us, total = 12.527 s Event stats: event_loop_lag_probe - 16795 total (0 active), Execution time: mean = 24.673 us, total = 414.377 ms, Queueing time: mean = 10.174 us, max = 13.097 ms, min = 1.994 us, total = 170.874 ms GcsInMemoryStore.Put - 6762 total (0 active), Execution time: mean = 46.973 us, total = 317.630 ms, Queueing time: mean = 335.267 us, max = 278.545 ms, min = 2.025 us, total = 2.267 s RayletLoadPulled - 4199 total (1 active), Execution time: mean = 248.851 us, total = 1.045 s, Queueing time: mean = 75.381 us, max = 472.090 us, min = -0.000 s, total = 316.524 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4196 total (0 active), Execution time: mean = 77.396 us, total = 324.755 ms, Queueing time: mean = 104.888 us, max = 769.233 us, min = 11.910 us, total = 440.112 ms NodeManagerService.grpc_client.GetResourceLoad - 4196 total (0 active), Execution time: mean = 1.495 ms, total = 6.272 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1690 total (0 active), Execution time: mean = 42.463 us, total = 71.762 ms, Queueing time: mean = 104.183 us, max = 228.583 us, min = 11.443 us, total = 176.069 ms InternalKVGcsService.grpc_server.InternalKVPut - 1690 total (0 active), Execution time: mean = 566.968 us, total = 958.175 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1679 total (0 active), Execution time: mean = 35.388 us, total = 59.417 ms, Queueing time: mean = 117.546 us, max = 1.498 ms, min = 20.329 us, total = 197.360 ms NodeInfoGcsService.grpc_server.CheckAlive - 1679 total (0 active), Execution time: mean = 593.731 us, total = 996.874 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1401 total (1 active), Execution time: mean = 9.613 us, total = 13.468 ms, Queueing time: mean = 76.012 us, max = 328.578 us, min = 16.354 us, total = 106.493 ms HealthCheck - 1398 total (0 active), Execution time: mean = 7.735 us, total = 10.814 ms, Queueing time: mean = 102.741 us, max = 399.856 us, min = 16.806 us, total = 143.631 ms GcsInMemoryStore.Get - 907 total (0 active), Execution time: mean = 21.524 us, total = 19.522 ms, Queueing time: mean = 13.312 us, max = 64.163 us, min = 3.701 us, total = 12.074 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 906 total (0 active), Execution time: mean = 56.229 us, total = 50.943 ms, Queueing time: mean = 106.432 us, max = 269.544 us, min = 13.461 us, total = 96.427 ms InternalKVGcsService.grpc_server.InternalKVGet - 906 total (0 active), Execution time: mean = 610.457 us, total = 553.074 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 838 total (0 active), Execution time: mean = 92.433 us, total = 77.459 ms, Queueing time: mean = 118.015 us, max = 296.469 us, min = 19.828 us, total = 98.897 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 838 total (0 active), Execution time: mean = 685.911 us, total = 574.794 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 420 total (1 active), Execution time: mean = 1.422 ms, total = 597.160 ms, Queueing time: mean = 71.975 us, max = 315.299 us, min = -0.000 s, total = 30.230 ms GCSServer.deadline_timer.debug_state_event_stats_print - 70 total (1 active, 1 running), Execution time: mean = 1.565 ms, total = 109.541 ms, Queueing time: mean = 65.008 us, max = 182.019 us, min = 12.907 us, total = 4.551 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:11:53,919 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 193869 total (1 active) Queueing time: mean = 49.254 us, max = 7.565 ms, min = 2.216 us, total = 9.549 s Execution time: mean = 253.530 us, total = 49.152 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 88116 total (0 active), Execution time: mean = 524.624 us, total = 46.228 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 88116 total (0 active), Execution time: mean = 27.531 us, total = 2.426 s, Queueing time: mean = 105.591 us, max = 7.565 ms, min = 3.372 us, total = 9.304 s event_loop_lag_probe - 16795 total (0 active), Execution time: mean = 29.062 us, total = 488.102 ms, Queueing time: mean = 10.498 us, max = 4.983 ms, min = 2.216 us, total = 176.317 ms GcsTaskManager.GcJobSummary - 841 total (1 active), Execution time: mean = 11.704 us, total = 9.843 ms, Queueing time: mean = 81.144 us, max = 229.756 us, min = 17.567 us, total = 68.242 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:11:53,919 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20707 total (25 active) Queueing time: mean = 19.595 us, max = 333.684 us, min = 2.330 us, total = 405.752 ms Execution time: mean = 4.835 s, total = 100119.995 s Event stats: event_loop_lag_probe - 16795 total (0 active), Execution time: mean = 31.707 us, total = 532.516 ms, Queueing time: mean = 11.664 us, max = 291.078 us, min = 2.330 us, total = 195.898 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1054 total (0 active), Execution time: mean = 43.164 us, total = 45.495 ms, Queueing time: mean = 104.540 us, max = 333.684 us, min = 8.944 us, total = 110.185 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1054 total (24 active), Execution time: mean = 94.989 s, total = 100118.722 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 869 total (0 active), Execution time: mean = 116.955 us, total = 101.634 ms, Queueing time: mean = 109.381 us, max = 247.252 us, min = 16.251 us, total = 95.052 ms InternalPubSubGcsService.grpc_server.GcsPublish - 869 total (0 active), Execution time: mean = 650.795 us, total = 565.541 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 678.308 us, total = 10.175 ms, Queueing time: mean = 79.940 us, max = 153.556 us, min = 48.361 us, total = 1.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:11:53,919 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 16822 total (0 active) Queueing time: mean = 10.098 us, max = 1.423 ms, min = 568.000 ns, total = 169.870 ms Execution time: mean = 27.132 us, total = 456.411 ms Event stats: event_loop_lag_probe - 16796 total (0 active), Execution time: mean = 27.082 us, total = 454.862 ms, Queueing time: mean = 10.002 us, max = 1.423 ms, min = 1.850 us, total = 167.991 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:12:53,919 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 850 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 882 - cumulative published bytes: 700619 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:12:53,920 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 51448 total (4 active) Queueing time: mean = 90.656 us, max = 279.303 ms, min = -0.000 s, total = 4.664 s Execution time: mean = 246.787 us, total = 12.697 s Event stats: event_loop_lag_probe - 17035 total (0 active), Execution time: mean = 24.657 us, total = 420.024 ms, Queueing time: mean = 10.153 us, max = 13.097 ms, min = 1.994 us, total = 172.958 ms GcsInMemoryStore.Put - 6858 total (0 active), Execution time: mean = 46.380 us, total = 318.072 ms, Queueing time: mean = 331.151 us, max = 278.545 ms, min = 2.025 us, total = 2.271 s RayletLoadPulled - 4259 total (1 active), Execution time: mean = 248.846 us, total = 1.060 s, Queueing time: mean = 75.230 us, max = 472.090 us, min = -0.000 s, total = 320.406 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4256 total (0 active), Execution time: mean = 77.374 us, total = 329.303 ms, Queueing time: mean = 104.493 us, max = 769.233 us, min = 11.910 us, total = 444.721 ms NodeManagerService.grpc_client.GetResourceLoad - 4256 total (0 active), Execution time: mean = 1.494 ms, total = 6.360 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1714 total (0 active), Execution time: mean = 42.472 us, total = 72.797 ms, Queueing time: mean = 103.830 us, max = 228.583 us, min = 11.443 us, total = 177.965 ms InternalKVGcsService.grpc_server.InternalKVPut - 1714 total (0 active), Execution time: mean = 566.510 us, total = 970.998 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1703 total (0 active), Execution time: mean = 35.527 us, total = 60.503 ms, Queueing time: mean = 117.240 us, max = 1.498 ms, min = 20.329 us, total = 199.659 ms NodeInfoGcsService.grpc_server.CheckAlive - 1703 total (0 active), Execution time: mean = 593.323 us, total = 1.010 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1421 total (1 active), Execution time: mean = 9.602 us, total = 13.644 ms, Queueing time: mean = 75.838 us, max = 328.578 us, min = 16.354 us, total = 107.767 ms HealthCheck - 1418 total (0 active), Execution time: mean = 7.722 us, total = 10.950 ms, Queueing time: mean = 102.443 us, max = 399.856 us, min = 16.806 us, total = 145.264 ms GcsInMemoryStore.Get - 919 total (0 active), Execution time: mean = 21.499 us, total = 19.758 ms, Queueing time: mean = 13.290 us, max = 64.163 us, min = 3.701 us, total = 12.214 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 918 total (0 active), Execution time: mean = 56.267 us, total = 51.653 ms, Queueing time: mean = 106.133 us, max = 269.544 us, min = 13.461 us, total = 97.430 ms InternalKVGcsService.grpc_server.InternalKVGet - 918 total (0 active), Execution time: mean = 610.067 us, total = 560.042 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 850 total (0 active), Execution time: mean = 92.589 us, total = 78.700 ms, Queueing time: mean = 117.670 us, max = 296.469 us, min = 19.828 us, total = 100.019 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 850 total (0 active), Execution time: mean = 685.292 us, total = 582.499 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 426 total (1 active), Execution time: mean = 1.423 ms, total = 606.257 ms, Queueing time: mean = 71.887 us, max = 315.299 us, min = -0.000 s, total = 30.624 ms GCSServer.deadline_timer.debug_state_event_stats_print - 71 total (1 active, 1 running), Execution time: mean = 1.567 ms, total = 111.245 ms, Queueing time: mean = 65.071 us, max = 182.019 us, min = 12.907 us, total = 4.620 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:12:53,921 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 196641 total (1 active) Queueing time: mean = 49.272 us, max = 7.565 ms, min = 2.065 us, total = 9.689 s Execution time: mean = 253.569 us, total = 49.862 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 89376 total (0 active), Execution time: mean = 524.696 us, total = 46.895 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 89376 total (0 active), Execution time: mean = 27.547 us, total = 2.462 s, Queueing time: mean = 105.629 us, max = 7.565 ms, min = 3.372 us, total = 9.441 s event_loop_lag_probe - 17035 total (0 active), Execution time: mean = 29.047 us, total = 494.824 ms, Queueing time: mean = 10.494 us, max = 4.983 ms, min = 2.065 us, total = 178.763 ms GcsTaskManager.GcJobSummary - 853 total (1 active), Execution time: mean = 11.719 us, total = 9.996 ms, Queueing time: mean = 81.305 us, max = 229.756 us, min = 17.567 us, total = 69.353 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:12:53,921 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 20999 total (25 active) Queueing time: mean = 19.591 us, max = 333.684 us, min = 2.330 us, total = 411.389 ms Execution time: mean = 4.771 s, total = 100180.191 s Event stats: event_loop_lag_probe - 17035 total (0 active), Execution time: mean = 31.703 us, total = 540.065 ms, Queueing time: mean = 11.665 us, max = 291.078 us, min = 2.330 us, total = 198.706 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1067 total (0 active), Execution time: mean = 43.297 us, total = 46.198 ms, Queueing time: mean = 104.613 us, max = 333.684 us, min = 8.944 us, total = 111.622 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1067 total (24 active), Execution time: mean = 93.888 s, total = 100178.901 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 882 total (0 active), Execution time: mean = 116.801 us, total = 103.019 ms, Queueing time: mean = 109.348 us, max = 247.252 us, min = 16.251 us, total = 96.445 ms InternalPubSubGcsService.grpc_server.GcsPublish - 882 total (0 active), Execution time: mean = 650.254 us, total = 573.524 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 678.308 us, total = 10.175 ms, Queueing time: mean = 79.940 us, max = 153.556 us, min = 48.361 us, total = 1.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:12:53,921 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17062 total (0 active) Queueing time: mean = 10.091 us, max = 1.423 ms, min = 568.000 ns, total = 172.176 ms Execution time: mean = 27.221 us, total = 464.440 ms Event stats: event_loop_lag_probe - 17036 total (0 active), Execution time: mean = 27.171 us, total = 462.891 ms, Queueing time: mean = 9.996 us, max = 1.423 ms, min = 1.850 us, total = 170.296 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:13:53,921 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 862 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 895 - cumulative published bytes: 710481 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:13:53,922 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 52167 total (4 active) Queueing time: mean = 89.999 us, max = 279.303 ms, min = -0.000 s, total = 4.695 s Execution time: mean = 246.802 us, total = 12.875 s Event stats: event_loop_lag_probe - 17275 total (0 active), Execution time: mean = 24.679 us, total = 426.328 ms, Queueing time: mean = 10.145 us, max = 13.097 ms, min = 1.994 us, total = 175.257 ms GcsInMemoryStore.Put - 6954 total (0 active), Execution time: mean = 45.813 us, total = 318.584 ms, Queueing time: mean = 327.221 us, max = 278.545 ms, min = 2.025 us, total = 2.275 s RayletLoadPulled - 4319 total (1 active), Execution time: mean = 248.907 us, total = 1.075 s, Queueing time: mean = 75.253 us, max = 472.090 us, min = -0.000 s, total = 325.016 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4316 total (0 active), Execution time: mean = 77.408 us, total = 334.093 ms, Queueing time: mean = 104.451 us, max = 769.233 us, min = 11.910 us, total = 450.808 ms NodeManagerService.grpc_client.GetResourceLoad - 4316 total (0 active), Execution time: mean = 1.495 ms, total = 6.452 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1738 total (0 active), Execution time: mean = 42.502 us, total = 73.869 ms, Queueing time: mean = 103.967 us, max = 228.583 us, min = 11.443 us, total = 180.695 ms InternalKVGcsService.grpc_server.InternalKVPut - 1738 total (0 active), Execution time: mean = 566.955 us, total = 985.368 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1727 total (0 active), Execution time: mean = 35.525 us, total = 61.352 ms, Queueing time: mean = 117.425 us, max = 1.498 ms, min = 20.329 us, total = 202.793 ms NodeInfoGcsService.grpc_server.CheckAlive - 1727 total (0 active), Execution time: mean = 593.708 us, total = 1.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1441 total (1 active), Execution time: mean = 9.601 us, total = 13.836 ms, Queueing time: mean = 76.015 us, max = 328.578 us, min = 16.354 us, total = 109.537 ms HealthCheck - 1438 total (0 active), Execution time: mean = 7.729 us, total = 11.114 ms, Queueing time: mean = 102.594 us, max = 399.856 us, min = 16.806 us, total = 147.530 ms GcsInMemoryStore.Get - 931 total (0 active), Execution time: mean = 21.470 us, total = 19.988 ms, Queueing time: mean = 13.288 us, max = 64.163 us, min = 3.701 us, total = 12.372 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 930 total (0 active), Execution time: mean = 56.303 us, total = 52.362 ms, Queueing time: mean = 106.326 us, max = 269.544 us, min = 13.461 us, total = 98.883 ms InternalKVGcsService.grpc_server.InternalKVGet - 930 total (0 active), Execution time: mean = 609.847 us, total = 567.158 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 862 total (0 active), Execution time: mean = 92.589 us, total = 79.812 ms, Queueing time: mean = 117.693 us, max = 296.469 us, min = 19.828 us, total = 101.451 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 862 total (0 active), Execution time: mean = 685.617 us, total = 591.001 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 432 total (1 active), Execution time: mean = 1.422 ms, total = 614.385 ms, Queueing time: mean = 71.895 us, max = 315.299 us, min = -0.000 s, total = 31.059 ms GCSServer.deadline_timer.debug_state_event_stats_print - 72 total (1 active, 1 running), Execution time: mean = 1.571 ms, total = 113.099 ms, Queueing time: mean = 65.127 us, max = 182.019 us, min = 12.907 us, total = 4.689 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:13:53,922 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 199412 total (1 active) Queueing time: mean = 49.313 us, max = 7.565 ms, min = 2.065 us, total = 9.834 s Execution time: mean = 253.700 us, total = 50.591 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 90636 total (0 active), Execution time: mean = 524.958 us, total = 47.580 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 90636 total (0 active), Execution time: mean = 27.571 us, total = 2.499 s, Queueing time: mean = 105.719 us, max = 7.565 ms, min = 3.372 us, total = 9.582 s event_loop_lag_probe - 17274 total (0 active), Execution time: mean = 29.048 us, total = 501.781 ms, Queueing time: mean = 10.494 us, max = 4.983 ms, min = 2.065 us, total = 181.278 ms GcsTaskManager.GcJobSummary - 865 total (1 active), Execution time: mean = 11.715 us, total = 10.133 ms, Queueing time: mean = 81.258 us, max = 229.756 us, min = 17.567 us, total = 70.288 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:13:53,922 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21290 total (25 active) Queueing time: mean = 19.585 us, max = 333.684 us, min = 2.330 us, total = 416.962 ms Execution time: mean = 4.708 s, total = 100240.290 s Event stats: event_loop_lag_probe - 17274 total (0 active), Execution time: mean = 31.711 us, total = 547.770 ms, Queueing time: mean = 11.671 us, max = 291.078 us, min = 2.330 us, total = 201.604 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1080 total (0 active), Execution time: mean = 43.311 us, total = 46.775 ms, Queueing time: mean = 104.507 us, max = 333.684 us, min = 8.944 us, total = 112.867 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1080 total (24 active), Execution time: mean = 92.814 s, total = 100238.982 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 895 total (0 active), Execution time: mean = 116.693 us, total = 104.441 ms, Queueing time: mean = 109.357 us, max = 247.252 us, min = 16.251 us, total = 97.874 ms InternalPubSubGcsService.grpc_server.GcsPublish - 895 total (0 active), Execution time: mean = 650.430 us, total = 582.135 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 678.308 us, total = 10.175 ms, Queueing time: mean = 79.940 us, max = 153.556 us, min = 48.361 us, total = 1.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:13:53,922 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17302 total (0 active) Queueing time: mean = 10.100 us, max = 1.423 ms, min = 568.000 ns, total = 174.755 ms Execution time: mean = 27.245 us, total = 471.401 ms Event stats: event_loop_lag_probe - 17276 total (0 active), Execution time: mean = 27.197 us, total = 469.852 ms, Queueing time: mean = 10.007 us, max = 1.423 ms, min = 1.850 us, total = 172.876 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:14:53,922 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 874 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 907 - cumulative published bytes: 720309 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:14:53,923 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 52886 total (4 active) Queueing time: mean = 89.348 us, max = 279.303 ms, min = -0.000 s, total = 4.725 s Execution time: mean = 246.965 us, total = 13.061 s Event stats: event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 24.700 us, total = 432.618 ms, Queueing time: mean = 10.137 us, max = 13.097 ms, min = 1.994 us, total = 177.553 ms GcsInMemoryStore.Put - 7050 total (0 active), Execution time: mean = 45.260 us, total = 319.085 ms, Queueing time: mean = 323.406 us, max = 278.545 ms, min = 2.025 us, total = 2.280 s RayletLoadPulled - 4379 total (1 active), Execution time: mean = 249.380 us, total = 1.092 s, Queueing time: mean = 75.306 us, max = 472.090 us, min = -0.000 s, total = 329.766 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4376 total (0 active), Execution time: mean = 77.410 us, total = 338.745 ms, Queueing time: mean = 104.306 us, max = 769.233 us, min = 11.910 us, total = 456.443 ms NodeManagerService.grpc_client.GetResourceLoad - 4376 total (0 active), Execution time: mean = 1.497 ms, total = 6.551 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1762 total (0 active), Execution time: mean = 42.537 us, total = 74.950 ms, Queueing time: mean = 104.040 us, max = 228.583 us, min = 11.443 us, total = 183.318 ms InternalKVGcsService.grpc_server.InternalKVPut - 1762 total (0 active), Execution time: mean = 567.357 us, total = 999.683 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1751 total (0 active), Execution time: mean = 35.671 us, total = 62.459 ms, Queueing time: mean = 117.608 us, max = 1.498 ms, min = 20.329 us, total = 205.932 ms NodeInfoGcsService.grpc_server.CheckAlive - 1751 total (0 active), Execution time: mean = 594.233 us, total = 1.041 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1461 total (1 active), Execution time: mean = 9.597 us, total = 14.022 ms, Queueing time: mean = 76.068 us, max = 328.578 us, min = 16.354 us, total = 111.135 ms HealthCheck - 1458 total (0 active), Execution time: mean = 7.738 us, total = 11.282 ms, Queueing time: mean = 102.731 us, max = 399.856 us, min = 16.806 us, total = 149.782 ms GcsInMemoryStore.Get - 943 total (0 active), Execution time: mean = 21.479 us, total = 20.254 ms, Queueing time: mean = 13.303 us, max = 64.163 us, min = 3.701 us, total = 12.544 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 942 total (0 active), Execution time: mean = 56.388 us, total = 53.117 ms, Queueing time: mean = 106.363 us, max = 269.544 us, min = 13.461 us, total = 100.194 ms InternalKVGcsService.grpc_server.InternalKVGet - 942 total (0 active), Execution time: mean = 609.713 us, total = 574.349 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 874 total (0 active), Execution time: mean = 92.801 us, total = 81.108 ms, Queueing time: mean = 117.916 us, max = 296.469 us, min = 19.828 us, total = 103.058 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 874 total (0 active), Execution time: mean = 686.257 us, total = 599.789 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 438 total (1 active), Execution time: mean = 1.420 ms, total = 621.807 ms, Queueing time: mean = 71.712 us, max = 315.299 us, min = -0.000 s, total = 31.410 ms GCSServer.deadline_timer.debug_state_event_stats_print - 73 total (1 active, 1 running), Execution time: mean = 1.561 ms, total = 113.938 ms, Queueing time: mean = 64.858 us, max = 182.019 us, min = 12.907 us, total = 4.735 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:14:53,923 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 202184 total (1 active) Queueing time: mean = 49.339 us, max = 7.565 ms, min = 2.065 us, total = 9.976 s Execution time: mean = 253.804 us, total = 51.315 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 91896 total (0 active), Execution time: mean = 525.131 us, total = 48.257 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 91896 total (0 active), Execution time: mean = 27.614 us, total = 2.538 s, Queueing time: mean = 105.772 us, max = 7.565 ms, min = 3.372 us, total = 9.720 s event_loop_lag_probe - 17514 total (0 active), Execution time: mean = 29.101 us, total = 509.679 ms, Queueing time: mean = 10.507 us, max = 4.983 ms, min = 2.065 us, total = 184.011 ms GcsTaskManager.GcJobSummary - 877 total (1 active), Execution time: mean = 11.746 us, total = 10.301 ms, Queueing time: mean = 81.463 us, max = 229.756 us, min = 17.567 us, total = 71.443 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:14:53,924 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21578 total (25 active) Queueing time: mean = 19.581 us, max = 333.684 us, min = 2.330 us, total = 422.524 ms Execution time: mean = 4.648 s, total = 100300.479 s Event stats: event_loop_lag_probe - 17514 total (0 active), Execution time: mean = 31.715 us, total = 555.454 ms, Queueing time: mean = 11.672 us, max = 291.078 us, min = 2.330 us, total = 204.425 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1092 total (0 active), Execution time: mean = 43.305 us, total = 47.289 ms, Queueing time: mean = 104.626 us, max = 333.684 us, min = 8.944 us, total = 114.252 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1092 total (24 active), Execution time: mean = 91.849 s, total = 100299.152 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 907 total (0 active), Execution time: mean = 116.981 us, total = 106.102 ms, Queueing time: mean = 109.405 us, max = 247.252 us, min = 16.251 us, total = 99.230 ms InternalPubSubGcsService.grpc_server.GcsPublish - 907 total (0 active), Execution time: mean = 651.261 us, total = 590.694 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 678.308 us, total = 10.175 ms, Queueing time: mean = 79.940 us, max = 153.556 us, min = 48.361 us, total = 1.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:14:53,924 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17541 total (0 active) Queueing time: mean = 10.107 us, max = 1.423 ms, min = 568.000 ns, total = 177.290 ms Execution time: mean = 27.263 us, total = 478.224 ms Event stats: event_loop_lag_probe - 17515 total (0 active), Execution time: mean = 27.215 us, total = 476.675 ms, Queueing time: mean = 10.015 us, max = 1.423 ms, min = 1.850 us, total = 175.411 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:15:53,924 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 886 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 920 - cumulative published bytes: 730171 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:15:53,925 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 53605 total (4 active) Queueing time: mean = 88.709 us, max = 279.303 ms, min = -0.000 s, total = 4.755 s Execution time: mean = 246.886 us, total = 13.234 s Event stats: event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 24.698 us, total = 438.519 ms, Queueing time: mean = 10.122 us, max = 13.097 ms, min = 1.994 us, total = 179.712 ms GcsInMemoryStore.Put - 7146 total (0 active), Execution time: mean = 44.737 us, total = 319.692 ms, Queueing time: mean = 319.647 us, max = 278.545 ms, min = 2.025 us, total = 2.284 s RayletLoadPulled - 4439 total (1 active), Execution time: mean = 249.263 us, total = 1.106 s, Queueing time: mean = 75.328 us, max = 472.090 us, min = -0.000 s, total = 334.380 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4436 total (0 active), Execution time: mean = 77.419 us, total = 343.432 ms, Queueing time: mean = 104.374 us, max = 769.233 us, min = 11.910 us, total = 463.004 ms NodeManagerService.grpc_client.GetResourceLoad - 4436 total (0 active), Execution time: mean = 1.497 ms, total = 6.639 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1786 total (0 active), Execution time: mean = 42.556 us, total = 76.005 ms, Queueing time: mean = 104.233 us, max = 228.583 us, min = 11.443 us, total = 186.160 ms InternalKVGcsService.grpc_server.InternalKVPut - 1786 total (0 active), Execution time: mean = 567.852 us, total = 1.014 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1775 total (0 active), Execution time: mean = 35.632 us, total = 63.247 ms, Queueing time: mean = 117.556 us, max = 1.498 ms, min = 20.329 us, total = 208.662 ms NodeInfoGcsService.grpc_server.CheckAlive - 1775 total (0 active), Execution time: mean = 594.391 us, total = 1.055 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1481 total (1 active), Execution time: mean = 9.603 us, total = 14.222 ms, Queueing time: mean = 75.973 us, max = 328.578 us, min = 16.354 us, total = 112.517 ms HealthCheck - 1478 total (0 active), Execution time: mean = 7.757 us, total = 11.465 ms, Queueing time: mean = 102.688 us, max = 399.856 us, min = 16.806 us, total = 151.772 ms GcsInMemoryStore.Get - 955 total (0 active), Execution time: mean = 21.486 us, total = 20.519 ms, Queueing time: mean = 13.314 us, max = 64.163 us, min = 3.701 us, total = 12.715 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 954 total (0 active), Execution time: mean = 56.446 us, total = 53.849 ms, Queueing time: mean = 106.399 us, max = 269.544 us, min = 13.461 us, total = 101.505 ms InternalKVGcsService.grpc_server.InternalKVGet - 954 total (0 active), Execution time: mean = 609.893 us, total = 581.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 886 total (0 active), Execution time: mean = 92.813 us, total = 82.233 ms, Queueing time: mean = 118.072 us, max = 296.469 us, min = 19.828 us, total = 104.612 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 886 total (0 active), Execution time: mean = 687.021 us, total = 608.700 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 444 total (1 active), Execution time: mean = 1.420 ms, total = 630.304 ms, Queueing time: mean = 71.652 us, max = 315.299 us, min = -0.000 s, total = 31.814 ms GCSServer.deadline_timer.debug_state_event_stats_print - 74 total (1 active, 1 running), Execution time: mean = 1.562 ms, total = 115.618 ms, Queueing time: mean = 64.871 us, max = 182.019 us, min = 12.907 us, total = 4.800 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:15:53,925 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 204956 total (1 active) Queueing time: mean = 49.374 us, max = 7.565 ms, min = 2.065 us, total = 10.119 s Execution time: mean = 254.011 us, total = 52.061 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 93156 total (0 active), Execution time: mean = 525.512 us, total = 48.955 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 93156 total (0 active), Execution time: mean = 27.691 us, total = 2.580 s, Queueing time: mean = 105.851 us, max = 7.565 ms, min = 3.372 us, total = 9.861 s event_loop_lag_probe - 17754 total (0 active), Execution time: mean = 29.087 us, total = 516.418 ms, Queueing time: mean = 10.498 us, max = 4.983 ms, min = 2.065 us, total = 186.380 ms GcsTaskManager.GcJobSummary - 889 total (1 active), Execution time: mean = 11.741 us, total = 10.438 ms, Queueing time: mean = 81.401 us, max = 229.756 us, min = 17.567 us, total = 72.366 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:15:53,925 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 21870 total (25 active) Queueing time: mean = 19.587 us, max = 333.684 us, min = 2.330 us, total = 428.360 ms Execution time: mean = 4.589 s, total = 100360.686 s Event stats: event_loop_lag_probe - 17754 total (0 active), Execution time: mean = 31.734 us, total = 563.409 ms, Queueing time: mean = 11.680 us, max = 291.078 us, min = 2.330 us, total = 207.362 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1105 total (0 active), Execution time: mean = 43.392 us, total = 47.949 ms, Queueing time: mean = 104.637 us, max = 333.684 us, min = 8.944 us, total = 115.624 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1105 total (24 active), Execution time: mean = 90.823 s, total = 100359.341 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 920 total (0 active), Execution time: mean = 117.075 us, total = 107.709 ms, Queueing time: mean = 109.519 us, max = 247.252 us, min = 16.251 us, total = 100.758 ms InternalPubSubGcsService.grpc_server.GcsPublish - 920 total (0 active), Execution time: mean = 651.229 us, total = 599.131 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 15 total (1 active), Execution time: mean = 678.308 us, total = 10.175 ms, Queueing time: mean = 79.940 us, max = 153.556 us, min = 48.361 us, total = 1.199 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:15:53,925 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 17781 total (0 active) Queueing time: mean = 10.095 us, max = 1.423 ms, min = 568.000 ns, total = 179.498 ms Execution time: mean = 27.241 us, total = 484.379 ms Event stats: event_loop_lag_probe - 17755 total (0 active), Execution time: mean = 27.194 us, total = 482.829 ms, Queueing time: mean = 10.004 us, max = 1.423 ms, min = 1.850 us, total = 177.619 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:16:53,926 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 898 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 932 - cumulative published bytes: 739999 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:16:53,926 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 54324 total (4 active) Queueing time: mean = 88.073 us, max = 279.303 ms, min = -0.000 s, total = 4.784 s Execution time: mean = 246.818 us, total = 13.408 s Event stats: event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 24.702 us, total = 444.506 ms, Queueing time: mean = 10.108 us, max = 13.097 ms, min = 1.994 us, total = 181.899 ms GcsInMemoryStore.Put - 7242 total (0 active), Execution time: mean = 44.219 us, total = 320.232 ms, Queueing time: mean = 316.031 us, max = 278.545 ms, min = 2.025 us, total = 2.289 s RayletLoadPulled - 4499 total (1 active), Execution time: mean = 249.349 us, total = 1.122 s, Queueing time: mean = 75.290 us, max = 472.090 us, min = -0.000 s, total = 338.729 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4496 total (0 active), Execution time: mean = 77.437 us, total = 348.157 ms, Queueing time: mean = 104.391 us, max = 769.233 us, min = 11.910 us, total = 469.340 ms NodeManagerService.grpc_client.GetResourceLoad - 4496 total (0 active), Execution time: mean = 1.497 ms, total = 6.729 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1810 total (0 active), Execution time: mean = 42.638 us, total = 77.175 ms, Queueing time: mean = 104.149 us, max = 228.583 us, min = 11.443 us, total = 188.511 ms InternalKVGcsService.grpc_server.InternalKVPut - 1810 total (0 active), Execution time: mean = 567.986 us, total = 1.028 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1799 total (0 active), Execution time: mean = 35.631 us, total = 64.099 ms, Queueing time: mean = 117.550 us, max = 1.498 ms, min = 20.329 us, total = 211.472 ms NodeInfoGcsService.grpc_server.CheckAlive - 1799 total (0 active), Execution time: mean = 594.590 us, total = 1.070 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1501 total (1 active), Execution time: mean = 9.602 us, total = 14.412 ms, Queueing time: mean = 76.026 us, max = 328.578 us, min = 16.354 us, total = 114.115 ms HealthCheck - 1498 total (0 active), Execution time: mean = 7.751 us, total = 11.611 ms, Queueing time: mean = 102.607 us, max = 399.856 us, min = 16.806 us, total = 153.705 ms GcsInMemoryStore.Get - 967 total (0 active), Execution time: mean = 21.472 us, total = 20.764 ms, Queueing time: mean = 13.329 us, max = 64.163 us, min = 3.701 us, total = 12.889 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 966 total (0 active), Execution time: mean = 56.457 us, total = 54.538 ms, Queueing time: mean = 106.414 us, max = 269.544 us, min = 13.461 us, total = 102.796 ms InternalKVGcsService.grpc_server.InternalKVGet - 966 total (0 active), Execution time: mean = 609.849 us, total = 589.114 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 898 total (0 active), Execution time: mean = 92.672 us, total = 83.219 ms, Queueing time: mean = 117.964 us, max = 296.469 us, min = 19.828 us, total = 105.932 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 898 total (0 active), Execution time: mean = 686.712 us, total = 616.667 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 450 total (1 active), Execution time: mean = 1.418 ms, total = 638.258 ms, Queueing time: mean = 71.398 us, max = 315.299 us, min = -0.000 s, total = 32.129 ms GCSServer.deadline_timer.debug_state_event_stats_print - 75 total (1 active, 1 running), Execution time: mean = 1.563 ms, total = 117.189 ms, Queueing time: mean = 64.953 us, max = 182.019 us, min = 12.907 us, total = 4.871 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:16:53,926 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 207728 total (1 active) Queueing time: mean = 49.387 us, max = 7.565 ms, min = 1.647 us, total = 10.259 s Execution time: mean = 254.125 us, total = 52.789 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 94416 total (0 active), Execution time: mean = 525.731 us, total = 49.637 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 94416 total (0 active), Execution time: mean = 27.728 us, total = 2.618 s, Queueing time: mean = 105.883 us, max = 7.565 ms, min = 3.372 us, total = 9.997 s event_loop_lag_probe - 17994 total (0 active), Execution time: mean = 29.062 us, total = 522.934 ms, Queueing time: mean = 10.483 us, max = 4.983 ms, min = 1.647 us, total = 188.640 ms GcsTaskManager.GcJobSummary - 901 total (1 active), Execution time: mean = 11.769 us, total = 10.604 ms, Queueing time: mean = 81.382 us, max = 229.756 us, min = 17.567 us, total = 73.326 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:16:53,927 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22163 total (25 active) Queueing time: mean = 19.581 us, max = 333.684 us, min = 2.330 us, total = 433.983 ms Execution time: mean = 4.585 s, total = 101620.906 s Event stats: event_loop_lag_probe - 17994 total (0 active), Execution time: mean = 31.739 us, total = 571.110 ms, Queueing time: mean = 11.676 us, max = 291.078 us, min = 2.330 us, total = 210.100 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1119 total (0 active), Execution time: mean = 43.382 us, total = 48.544 ms, Queueing time: mean = 104.702 us, max = 333.684 us, min = 8.944 us, total = 117.162 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1119 total (24 active), Execution time: mean = 90.813 s, total = 101619.543 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 932 total (0 active), Execution time: mean = 117.099 us, total = 109.136 ms, Queueing time: mean = 109.477 us, max = 247.252 us, min = 16.251 us, total = 102.033 ms InternalPubSubGcsService.grpc_server.GcsPublish - 932 total (0 active), Execution time: mean = 650.913 us, total = 606.651 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 640.528 us, total = 10.248 ms, Queueing time: mean = 79.400 us, max = 153.556 us, min = 48.361 us, total = 1.270 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:16:53,927 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18021 total (0 active) Queueing time: mean = 10.086 us, max = 1.423 ms, min = 568.000 ns, total = 181.762 ms Execution time: mean = 27.230 us, total = 490.708 ms Event stats: event_loop_lag_probe - 17995 total (0 active), Execution time: mean = 27.183 us, total = 489.159 ms, Queueing time: mean = 9.996 us, max = 1.423 ms, min = 1.850 us, total = 179.883 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:17:53,927 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 910 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 944 - cumulative published bytes: 749827 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:17:53,928 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 55043 total (4 active) Queueing time: mean = 87.396 us, max = 279.303 ms, min = -0.000 s, total = 4.811 s Execution time: mean = 246.765 us, total = 13.583 s Event stats: event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 24.709 us, total = 450.571 ms, Queueing time: mean = 10.098 us, max = 13.097 ms, min = 1.994 us, total = 184.135 ms GcsInMemoryStore.Put - 7338 total (0 active), Execution time: mean = 43.703 us, total = 320.693 ms, Queueing time: mean = 312.437 us, max = 278.545 ms, min = 2.025 us, total = 2.293 s RayletLoadPulled - 4559 total (1 active), Execution time: mean = 249.195 us, total = 1.136 s, Queueing time: mean = 75.214 us, max = 472.090 us, min = -0.000 s, total = 342.900 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4556 total (0 active), Execution time: mean = 77.381 us, total = 352.546 ms, Queueing time: mean = 104.130 us, max = 769.233 us, min = 11.910 us, total = 474.416 ms NodeManagerService.grpc_client.GetResourceLoad - 4556 total (0 active), Execution time: mean = 1.497 ms, total = 6.821 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1834 total (0 active), Execution time: mean = 42.668 us, total = 78.253 ms, Queueing time: mean = 103.793 us, max = 228.583 us, min = 11.443 us, total = 190.355 ms InternalKVGcsService.grpc_server.InternalKVPut - 1834 total (0 active), Execution time: mean = 567.667 us, total = 1.041 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1823 total (0 active), Execution time: mean = 35.600 us, total = 64.898 ms, Queueing time: mean = 117.624 us, max = 1.498 ms, min = 20.329 us, total = 214.428 ms NodeInfoGcsService.grpc_server.CheckAlive - 1823 total (0 active), Execution time: mean = 594.903 us, total = 1.085 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1521 total (1 active), Execution time: mean = 9.605 us, total = 14.609 ms, Queueing time: mean = 75.976 us, max = 328.578 us, min = 16.354 us, total = 115.560 ms HealthCheck - 1518 total (0 active), Execution time: mean = 7.746 us, total = 11.758 ms, Queueing time: mean = 102.365 us, max = 399.856 us, min = 16.806 us, total = 155.391 ms GcsInMemoryStore.Get - 979 total (0 active), Execution time: mean = 21.431 us, total = 20.981 ms, Queueing time: mean = 13.300 us, max = 64.163 us, min = 3.701 us, total = 13.020 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 978 total (0 active), Execution time: mean = 56.402 us, total = 55.161 ms, Queueing time: mean = 106.002 us, max = 269.544 us, min = 13.461 us, total = 103.670 ms InternalKVGcsService.grpc_server.InternalKVGet - 978 total (0 active), Execution time: mean = 609.029 us, total = 595.630 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 910 total (0 active), Execution time: mean = 92.618 us, total = 84.282 ms, Queueing time: mean = 117.672 us, max = 296.469 us, min = 19.828 us, total = 107.081 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 910 total (0 active), Execution time: mean = 686.113 us, total = 624.363 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 456 total (1 active), Execution time: mean = 1.420 ms, total = 647.502 ms, Queueing time: mean = 71.367 us, max = 315.299 us, min = -0.000 s, total = 32.543 ms GCSServer.deadline_timer.debug_state_event_stats_print - 76 total (1 active, 1 running), Execution time: mean = 1.557 ms, total = 118.351 ms, Queueing time: mean = 65.692 us, max = 182.019 us, min = 12.907 us, total = 4.993 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:17:53,928 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 210500 total (1 active) Queueing time: mean = 49.405 us, max = 7.565 ms, min = 1.647 us, total = 10.400 s Execution time: mean = 254.182 us, total = 53.505 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 95676 total (0 active), Execution time: mean = 525.842 us, total = 50.310 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 95676 total (0 active), Execution time: mean = 27.742 us, total = 2.654 s, Queueing time: mean = 105.924 us, max = 7.565 ms, min = 3.372 us, total = 10.134 s event_loop_lag_probe - 18234 total (0 active), Execution time: mean = 29.061 us, total = 529.893 ms, Queueing time: mean = 10.477 us, max = 4.983 ms, min = 1.647 us, total = 191.031 ms GcsTaskManager.GcJobSummary - 913 total (1 active), Execution time: mean = 11.743 us, total = 10.722 ms, Queueing time: mean = 81.340 us, max = 229.756 us, min = 17.567 us, total = 74.263 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:17:53,928 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22451 total (25 active) Queueing time: mean = 19.564 us, max = 333.684 us, min = 2.330 us, total = 439.229 ms Execution time: mean = 4.529 s, total = 101681.003 s Event stats: event_loop_lag_probe - 18234 total (0 active), Execution time: mean = 31.712 us, total = 578.240 ms, Queueing time: mean = 11.663 us, max = 291.078 us, min = 2.330 us, total = 212.659 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1131 total (0 active), Execution time: mean = 43.486 us, total = 49.183 ms, Queueing time: mean = 104.855 us, max = 333.684 us, min = 8.944 us, total = 118.591 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1131 total (24 active), Execution time: mean = 89.902 s, total = 101679.624 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 944 total (0 active), Execution time: mean = 116.951 us, total = 110.402 ms, Queueing time: mean = 109.418 us, max = 247.252 us, min = 16.251 us, total = 103.291 ms InternalPubSubGcsService.grpc_server.GcsPublish - 944 total (0 active), Execution time: mean = 650.551 us, total = 614.120 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 640.528 us, total = 10.248 ms, Queueing time: mean = 79.400 us, max = 153.556 us, min = 48.361 us, total = 1.270 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:17:53,928 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18261 total (0 active) Queueing time: mean = 10.086 us, max = 1.423 ms, min = 568.000 ns, total = 184.181 ms Execution time: mean = 27.234 us, total = 497.325 ms Event stats: event_loop_lag_probe - 18235 total (0 active), Execution time: mean = 27.188 us, total = 495.776 ms, Queueing time: mean = 9.997 us, max = 1.423 ms, min = 1.850 us, total = 182.301 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:18:53,929 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 922 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 956 - cumulative published bytes: 759655 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:18:53,929 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 55762 total (4 active) Queueing time: mean = 86.788 us, max = 279.303 ms, min = -0.000 s, total = 4.839 s Execution time: mean = 246.762 us, total = 13.760 s Event stats: event_loop_lag_probe - 18475 total (0 active), Execution time: mean = 24.710 us, total = 456.513 ms, Queueing time: mean = 10.087 us, max = 13.097 ms, min = 1.994 us, total = 186.367 ms GcsInMemoryStore.Put - 7434 total (0 active), Execution time: mean = 43.208 us, total = 321.212 ms, Queueing time: mean = 308.977 us, max = 278.545 ms, min = 2.025 us, total = 2.297 s RayletLoadPulled - 4619 total (1 active), Execution time: mean = 249.291 us, total = 1.151 s, Queueing time: mean = 75.178 us, max = 472.090 us, min = -0.000 s, total = 347.247 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4616 total (0 active), Execution time: mean = 77.422 us, total = 357.380 ms, Queueing time: mean = 103.995 us, max = 769.233 us, min = 11.910 us, total = 480.043 ms NodeManagerService.grpc_client.GetResourceLoad - 4616 total (0 active), Execution time: mean = 1.497 ms, total = 6.911 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1858 total (0 active), Execution time: mean = 42.878 us, total = 79.667 ms, Queueing time: mean = 103.825 us, max = 228.583 us, min = 11.443 us, total = 192.907 ms InternalKVGcsService.grpc_server.InternalKVPut - 1858 total (0 active), Execution time: mean = 567.964 us, total = 1.055 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1847 total (0 active), Execution time: mean = 35.702 us, total = 65.942 ms, Queueing time: mean = 117.616 us, max = 1.498 ms, min = 20.329 us, total = 217.236 ms NodeInfoGcsService.grpc_server.CheckAlive - 1847 total (0 active), Execution time: mean = 595.119 us, total = 1.099 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1541 total (1 active), Execution time: mean = 9.599 us, total = 14.792 ms, Queueing time: mean = 75.956 us, max = 328.578 us, min = 16.354 us, total = 117.048 ms HealthCheck - 1538 total (0 active), Execution time: mean = 7.747 us, total = 11.915 ms, Queueing time: mean = 102.380 us, max = 399.856 us, min = 16.806 us, total = 157.460 ms GcsInMemoryStore.Get - 991 total (0 active), Execution time: mean = 21.410 us, total = 21.218 ms, Queueing time: mean = 13.312 us, max = 64.163 us, min = 3.701 us, total = 13.193 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 990 total (0 active), Execution time: mean = 56.663 us, total = 56.097 ms, Queueing time: mean = 105.977 us, max = 269.544 us, min = 13.461 us, total = 104.917 ms InternalKVGcsService.grpc_server.InternalKVGet - 990 total (0 active), Execution time: mean = 608.935 us, total = 602.846 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 922 total (0 active), Execution time: mean = 92.873 us, total = 85.629 ms, Queueing time: mean = 117.900 us, max = 296.469 us, min = 19.828 us, total = 108.704 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 922 total (0 active), Execution time: mean = 686.561 us, total = 633.009 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 462 total (1 active), Execution time: mean = 1.421 ms, total = 656.352 ms, Queueing time: mean = 71.362 us, max = 315.299 us, min = -0.000 s, total = 32.969 ms GCSServer.deadline_timer.debug_state_event_stats_print - 77 total (1 active, 1 running), Execution time: mean = 1.560 ms, total = 120.084 ms, Queueing time: mean = 65.623 us, max = 182.019 us, min = 12.907 us, total = 5.053 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:18:53,929 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 213272 total (1 active) Queueing time: mean = 49.441 us, max = 7.565 ms, min = 1.647 us, total = 10.544 s Execution time: mean = 254.334 us, total = 54.242 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 96936 total (0 active), Execution time: mean = 526.134 us, total = 51.001 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 96936 total (0 active), Execution time: mean = 27.783 us, total = 2.693 s, Queueing time: mean = 106.003 us, max = 7.565 ms, min = 3.372 us, total = 10.275 s event_loop_lag_probe - 18474 total (0 active), Execution time: mean = 29.067 us, total = 536.992 ms, Queueing time: mean = 10.474 us, max = 4.983 ms, min = 1.647 us, total = 193.503 ms GcsTaskManager.GcJobSummary - 925 total (1 active), Execution time: mean = 11.755 us, total = 10.873 ms, Queueing time: mean = 81.369 us, max = 229.756 us, min = 17.567 us, total = 75.266 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:18:53,929 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 22739 total (25 active) Queueing time: mean = 19.557 us, max = 333.684 us, min = 2.330 us, total = 444.711 ms Execution time: mean = 4.474 s, total = 101741.188 s Event stats: event_loop_lag_probe - 18474 total (0 active), Execution time: mean = 31.716 us, total = 585.914 ms, Queueing time: mean = 11.663 us, max = 291.078 us, min = 2.330 us, total = 215.471 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1143 total (0 active), Execution time: mean = 43.540 us, total = 49.766 ms, Queueing time: mean = 104.941 us, max = 333.684 us, min = 8.944 us, total = 119.948 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1143 total (24 active), Execution time: mean = 89.011 s, total = 101739.791 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 956 total (0 active), Execution time: mean = 116.950 us, total = 111.804 ms, Queueing time: mean = 109.419 us, max = 247.252 us, min = 16.251 us, total = 104.604 ms InternalPubSubGcsService.grpc_server.GcsPublish - 956 total (0 active), Execution time: mean = 650.566 us, total = 621.941 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 640.528 us, total = 10.248 ms, Queueing time: mean = 79.400 us, max = 153.556 us, min = 48.361 us, total = 1.270 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:18:53,929 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18501 total (0 active) Queueing time: mean = 10.074 us, max = 1.423 ms, min = 568.000 ns, total = 186.379 ms Execution time: mean = 27.217 us, total = 503.549 ms Event stats: event_loop_lag_probe - 18475 total (0 active), Execution time: mean = 27.172 us, total = 502.000 ms, Queueing time: mean = 9.986 us, max = 1.423 ms, min = 1.850 us, total = 184.500 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:19:53,930 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 934 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 968 - cumulative published bytes: 769483 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:19:53,931 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 56481 total (4 active) Queueing time: mean = 86.192 us, max = 279.303 ms, min = -0.000 s, total = 4.868 s Execution time: mean = 246.692 us, total = 13.933 s Event stats: event_loop_lag_probe - 18715 total (0 active), Execution time: mean = 24.704 us, total = 462.331 ms, Queueing time: mean = 10.073 us, max = 13.097 ms, min = 1.994 us, total = 188.525 ms GcsInMemoryStore.Put - 7530 total (0 active), Execution time: mean = 42.733 us, total = 321.777 ms, Queueing time: mean = 305.580 us, max = 278.545 ms, min = 2.025 us, total = 2.301 s RayletLoadPulled - 4679 total (1 active), Execution time: mean = 249.298 us, total = 1.166 s, Queueing time: mean = 75.185 us, max = 472.090 us, min = -0.000 s, total = 351.790 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4676 total (0 active), Execution time: mean = 77.396 us, total = 361.903 ms, Queueing time: mean = 103.871 us, max = 769.233 us, min = 11.910 us, total = 485.701 ms NodeManagerService.grpc_client.GetResourceLoad - 4676 total (0 active), Execution time: mean = 1.497 ms, total = 7.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1882 total (0 active), Execution time: mean = 42.992 us, total = 80.910 ms, Queueing time: mean = 103.812 us, max = 228.583 us, min = 11.443 us, total = 195.373 ms InternalKVGcsService.grpc_server.InternalKVPut - 1882 total (0 active), Execution time: mean = 568.284 us, total = 1.070 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1871 total (0 active), Execution time: mean = 35.711 us, total = 66.815 ms, Queueing time: mean = 117.716 us, max = 1.498 ms, min = 20.329 us, total = 220.247 ms NodeInfoGcsService.grpc_server.CheckAlive - 1871 total (0 active), Execution time: mean = 595.552 us, total = 1.114 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1561 total (1 active), Execution time: mean = 9.593 us, total = 14.975 ms, Queueing time: mean = 75.886 us, max = 328.578 us, min = 16.354 us, total = 118.458 ms HealthCheck - 1558 total (0 active), Execution time: mean = 7.750 us, total = 12.074 ms, Queueing time: mean = 102.431 us, max = 399.856 us, min = 16.806 us, total = 159.587 ms GcsInMemoryStore.Get - 1003 total (0 active), Execution time: mean = 21.418 us, total = 21.482 ms, Queueing time: mean = 13.316 us, max = 64.163 us, min = 3.701 us, total = 13.356 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1002 total (0 active), Execution time: mean = 56.805 us, total = 56.919 ms, Queueing time: mean = 106.100 us, max = 269.544 us, min = 13.461 us, total = 106.312 ms InternalKVGcsService.grpc_server.InternalKVGet - 1002 total (0 active), Execution time: mean = 609.480 us, total = 610.699 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 934 total (0 active), Execution time: mean = 92.843 us, total = 86.715 ms, Queueing time: mean = 117.885 us, max = 296.469 us, min = 19.828 us, total = 110.104 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 934 total (0 active), Execution time: mean = 686.518 us, total = 641.208 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 468 total (1 active), Execution time: mean = 1.419 ms, total = 664.129 ms, Queueing time: mean = 71.118 us, max = 315.299 us, min = -0.000 s, total = 33.283 ms GCSServer.deadline_timer.debug_state_event_stats_print - 78 total (1 active, 1 running), Execution time: mean = 1.553 ms, total = 121.112 ms, Queueing time: mean = 65.112 us, max = 182.019 us, min = 12.907 us, total = 5.079 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:19:53,931 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 216044 total (1 active) Queueing time: mean = 49.492 us, max = 7.565 ms, min = 1.647 us, total = 10.692 s Execution time: mean = 254.532 us, total = 54.990 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 98196 total (0 active), Execution time: mean = 526.542 us, total = 51.704 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 98196 total (0 active), Execution time: mean = 27.814 us, total = 2.731 s, Queueing time: mean = 106.118 us, max = 7.565 ms, min = 3.372 us, total = 10.420 s event_loop_lag_probe - 18714 total (0 active), Execution time: mean = 29.046 us, total = 543.574 ms, Queueing time: mean = 10.465 us, max = 4.983 ms, min = 1.647 us, total = 195.847 ms GcsTaskManager.GcJobSummary - 937 total (1 active), Execution time: mean = 11.725 us, total = 10.987 ms, Queueing time: mean = 81.230 us, max = 229.756 us, min = 16.725 us, total = 76.112 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:19:53,931 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23027 total (25 active) Queueing time: mean = 19.548 us, max = 333.684 us, min = 2.330 us, total = 450.141 ms Execution time: mean = 4.421 s, total = 101801.446 s Event stats: event_loop_lag_probe - 18714 total (0 active), Execution time: mean = 31.721 us, total = 593.622 ms, Queueing time: mean = 11.666 us, max = 291.078 us, min = 2.330 us, total = 218.315 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1155 total (0 active), Execution time: mean = 43.606 us, total = 50.365 ms, Queueing time: mean = 104.932 us, max = 333.684 us, min = 8.944 us, total = 121.196 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1155 total (24 active), Execution time: mean = 88.139 s, total = 101800.031 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 968 total (0 active), Execution time: mean = 117.002 us, total = 113.258 ms, Queueing time: mean = 109.444 us, max = 247.252 us, min = 16.251 us, total = 105.941 ms InternalPubSubGcsService.grpc_server.GcsPublish - 968 total (0 active), Execution time: mean = 651.111 us, total = 630.275 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 640.528 us, total = 10.248 ms, Queueing time: mean = 79.400 us, max = 153.556 us, min = 48.361 us, total = 1.270 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:19:53,931 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18741 total (0 active) Queueing time: mean = 10.060 us, max = 1.423 ms, min = 568.000 ns, total = 188.538 ms Execution time: mean = 27.190 us, total = 509.562 ms Event stats: event_loop_lag_probe - 18715 total (0 active), Execution time: mean = 27.145 us, total = 508.013 ms, Queueing time: mean = 9.974 us, max = 1.423 ms, min = 1.850 us, total = 186.659 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:20:53,931 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 946 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 980 - cumulative published bytes: 779311 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:20:53,932 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 57200 total (4 active) Queueing time: mean = 85.658 us, max = 279.303 ms, min = -0.000 s, total = 4.900 s Execution time: mean = 246.672 us, total = 14.110 s Event stats: event_loop_lag_probe - 18955 total (0 active), Execution time: mean = 24.717 us, total = 468.512 ms, Queueing time: mean = 10.067 us, max = 13.097 ms, min = 1.994 us, total = 190.811 ms GcsInMemoryStore.Put - 7626 total (0 active), Execution time: mean = 42.262 us, total = 322.289 ms, Queueing time: mean = 302.352 us, max = 278.545 ms, min = 2.025 us, total = 2.306 s RayletLoadPulled - 4739 total (1 active), Execution time: mean = 249.420 us, total = 1.182 s, Queueing time: mean = 75.211 us, max = 472.090 us, min = -0.000 s, total = 356.425 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4736 total (0 active), Execution time: mean = 77.431 us, total = 366.712 ms, Queueing time: mean = 103.947 us, max = 769.233 us, min = 11.910 us, total = 492.294 ms NodeManagerService.grpc_client.GetResourceLoad - 4736 total (0 active), Execution time: mean = 1.497 ms, total = 7.089 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1906 total (0 active), Execution time: mean = 43.039 us, total = 82.033 ms, Queueing time: mean = 104.060 us, max = 228.583 us, min = 11.443 us, total = 198.338 ms InternalKVGcsService.grpc_server.InternalKVPut - 1906 total (0 active), Execution time: mean = 568.729 us, total = 1.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1895 total (0 active), Execution time: mean = 35.764 us, total = 67.773 ms, Queueing time: mean = 117.757 us, max = 1.498 ms, min = 20.329 us, total = 223.150 ms NodeInfoGcsService.grpc_server.CheckAlive - 1895 total (0 active), Execution time: mean = 595.847 us, total = 1.129 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1581 total (1 active), Execution time: mean = 9.583 us, total = 15.151 ms, Queueing time: mean = 75.765 us, max = 328.578 us, min = 16.354 us, total = 119.784 ms HealthCheck - 1578 total (0 active), Execution time: mean = 7.756 us, total = 12.238 ms, Queueing time: mean = 102.571 us, max = 399.856 us, min = 16.806 us, total = 161.857 ms GcsInMemoryStore.Get - 1015 total (0 active), Execution time: mean = 21.417 us, total = 21.738 ms, Queueing time: mean = 13.326 us, max = 64.163 us, min = 3.701 us, total = 13.526 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1014 total (0 active), Execution time: mean = 56.882 us, total = 57.678 ms, Queueing time: mean = 106.357 us, max = 269.544 us, min = 13.461 us, total = 107.846 ms InternalKVGcsService.grpc_server.InternalKVGet - 1014 total (0 active), Execution time: mean = 609.959 us, total = 618.499 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 946 total (0 active), Execution time: mean = 92.900 us, total = 87.883 ms, Queueing time: mean = 118.048 us, max = 296.469 us, min = 19.828 us, total = 111.674 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 946 total (0 active), Execution time: mean = 686.884 us, total = 649.792 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 474 total (1 active), Execution time: mean = 1.419 ms, total = 672.666 ms, Queueing time: mean = 71.047 us, max = 315.299 us, min = -0.000 s, total = 33.676 ms GCSServer.deadline_timer.debug_state_event_stats_print - 79 total (1 active, 1 running), Execution time: mean = 1.555 ms, total = 122.870 ms, Queueing time: mean = 65.143 us, max = 182.019 us, min = 12.907 us, total = 5.146 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:20:53,933 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 218816 total (1 active) Queueing time: mean = 49.520 us, max = 7.565 ms, min = 1.647 us, total = 10.836 s Execution time: mean = 254.697 us, total = 55.732 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 99456 total (0 active), Execution time: mean = 526.873 us, total = 52.401 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 99456 total (0 active), Execution time: mean = 27.842 us, total = 2.769 s, Queueing time: mean = 106.179 us, max = 7.565 ms, min = 3.372 us, total = 10.560 s event_loop_lag_probe - 18954 total (0 active), Execution time: mean = 29.058 us, total = 550.770 ms, Queueing time: mean = 10.469 us, max = 4.983 ms, min = 1.647 us, total = 198.427 ms GcsTaskManager.GcJobSummary - 949 total (1 active), Execution time: mean = 11.749 us, total = 11.150 ms, Queueing time: mean = 81.294 us, max = 229.756 us, min = 16.725 us, total = 77.148 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:20:53,933 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23315 total (25 active) Queueing time: mean = 19.549 us, max = 333.684 us, min = 2.330 us, total = 455.775 ms Execution time: mean = 4.369 s, total = 101861.643 s Event stats: event_loop_lag_probe - 18954 total (0 active), Execution time: mean = 31.712 us, total = 601.077 ms, Queueing time: mean = 11.664 us, max = 291.078 us, min = 2.330 us, total = 221.074 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1167 total (0 active), Execution time: mean = 43.740 us, total = 51.044 ms, Queueing time: mean = 105.056 us, max = 333.684 us, min = 8.944 us, total = 122.600 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1167 total (24 active), Execution time: mean = 87.284 s, total = 101860.210 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 980 total (0 active), Execution time: mean = 116.988 us, total = 114.648 ms, Queueing time: mean = 109.606 us, max = 247.252 us, min = 16.251 us, total = 107.414 ms InternalPubSubGcsService.grpc_server.GcsPublish - 980 total (0 active), Execution time: mean = 651.503 us, total = 638.473 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 16 total (1 active), Execution time: mean = 640.528 us, total = 10.248 ms, Queueing time: mean = 79.400 us, max = 153.556 us, min = 48.361 us, total = 1.270 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:20:53,933 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 18981 total (0 active) Queueing time: mean = 10.058 us, max = 1.423 ms, min = 568.000 ns, total = 190.908 ms Execution time: mean = 27.197 us, total = 516.218 ms Event stats: event_loop_lag_probe - 18955 total (0 active), Execution time: mean = 27.152 us, total = 514.669 ms, Queueing time: mean = 9.972 us, max = 1.423 ms, min = 1.850 us, total = 189.029 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:21:53,933 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 958 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 992 - cumulative published bytes: 789139 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:21:53,934 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 57918 total (4 active) Queueing time: mean = 85.128 us, max = 279.303 ms, min = -0.000 s, total = 4.930 s Execution time: mean = 246.668 us, total = 14.287 s Event stats: event_loop_lag_probe - 19194 total (0 active), Execution time: mean = 24.723 us, total = 474.532 ms, Queueing time: mean = 10.056 us, max = 13.097 ms, min = 1.994 us, total = 193.020 ms GcsInMemoryStore.Put - 7722 total (0 active), Execution time: mean = 41.803 us, total = 322.802 ms, Queueing time: mean = 299.103 us, max = 278.545 ms, min = 2.025 us, total = 2.310 s RayletLoadPulled - 4799 total (1 active), Execution time: mean = 249.460 us, total = 1.197 s, Queueing time: mean = 75.266 us, max = 472.090 us, min = -0.000 s, total = 361.202 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4796 total (0 active), Execution time: mean = 77.425 us, total = 371.332 ms, Queueing time: mean = 103.994 us, max = 769.233 us, min = 11.910 us, total = 498.755 ms NodeManagerService.grpc_client.GetResourceLoad - 4796 total (0 active), Execution time: mean = 1.497 ms, total = 7.179 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1930 total (0 active), Execution time: mean = 42.992 us, total = 82.975 ms, Queueing time: mean = 104.270 us, max = 228.583 us, min = 11.443 us, total = 201.241 ms InternalKVGcsService.grpc_server.InternalKVPut - 1930 total (0 active), Execution time: mean = 568.881 us, total = 1.098 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1919 total (0 active), Execution time: mean = 35.731 us, total = 68.568 ms, Queueing time: mean = 117.759 us, max = 1.498 ms, min = 20.329 us, total = 225.979 ms NodeInfoGcsService.grpc_server.CheckAlive - 1919 total (0 active), Execution time: mean = 595.958 us, total = 1.144 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1601 total (1 active), Execution time: mean = 9.581 us, total = 15.339 ms, Queueing time: mean = 75.842 us, max = 328.578 us, min = 16.354 us, total = 121.424 ms HealthCheck - 1598 total (0 active), Execution time: mean = 7.754 us, total = 12.391 ms, Queueing time: mean = 102.756 us, max = 399.856 us, min = 16.806 us, total = 164.204 ms GcsInMemoryStore.Get - 1027 total (0 active), Execution time: mean = 21.410 us, total = 21.989 ms, Queueing time: mean = 13.333 us, max = 64.163 us, min = 3.701 us, total = 13.693 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1026 total (0 active), Execution time: mean = 56.845 us, total = 58.323 ms, Queueing time: mean = 106.480 us, max = 269.544 us, min = 13.461 us, total = 109.249 ms InternalKVGcsService.grpc_server.InternalKVGet - 1026 total (0 active), Execution time: mean = 610.117 us, total = 625.980 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 958 total (0 active), Execution time: mean = 93.082 us, total = 89.172 ms, Queueing time: mean = 118.198 us, max = 296.469 us, min = 19.828 us, total = 113.234 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 958 total (0 active), Execution time: mean = 687.487 us, total = 658.613 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 480 total (1 active), Execution time: mean = 1.421 ms, total = 682.123 ms, Queueing time: mean = 71.165 us, max = 315.299 us, min = -0.000 s, total = 34.159 ms GCSServer.deadline_timer.debug_state_event_stats_print - 80 total (1 active, 1 running), Execution time: mean = 1.556 ms, total = 124.480 ms, Queueing time: mean = 65.283 us, max = 182.019 us, min = 12.907 us, total = 5.223 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:21:53,934 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 221588 total (1 active) Queueing time: mean = 49.553 us, max = 7.565 ms, min = 1.647 us, total = 10.980 s Execution time: mean = 254.783 us, total = 56.457 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 100716 total (0 active), Execution time: mean = 527.047 us, total = 53.082 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 100716 total (0 active), Execution time: mean = 27.858 us, total = 2.806 s, Queueing time: mean = 106.250 us, max = 7.565 ms, min = 3.372 us, total = 10.701 s event_loop_lag_probe - 19194 total (0 active), Execution time: mean = 29.059 us, total = 557.753 ms, Queueing time: mean = 10.467 us, max = 4.983 ms, min = 1.647 us, total = 200.901 ms GcsTaskManager.GcJobSummary - 961 total (1 active), Execution time: mean = 11.743 us, total = 11.285 ms, Queueing time: mean = 81.560 us, max = 229.756 us, min = 16.725 us, total = 78.379 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:21:53,935 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23646 total (25 active) Queueing time: mean = 19.615 us, max = 333.684 us, min = 2.330 us, total = 463.826 ms Execution time: mean = 4.843 s, total = 114521.739 s Event stats: event_loop_lag_probe - 19194 total (0 active), Execution time: mean = 31.720 us, total = 608.826 ms, Queueing time: mean = 11.667 us, max = 291.078 us, min = 2.330 us, total = 223.927 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1200 total (0 active), Execution time: mean = 43.307 us, total = 51.968 ms, Queueing time: mean = 104.903 us, max = 333.684 us, min = 6.593 us, total = 125.884 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1200 total (24 active), Execution time: mean = 95.434 s, total = 114520.287 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 992 total (0 active), Execution time: mean = 116.914 us, total = 115.979 ms, Queueing time: mean = 110.120 us, max = 249.252 us, min = 16.251 us, total = 109.239 ms InternalPubSubGcsService.grpc_server.GcsPublish - 992 total (0 active), Execution time: mean = 652.440 us, total = 647.221 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 648.681 us, total = 11.028 ms, Queueing time: mean = 79.898 us, max = 153.556 us, min = 48.361 us, total = 1.358 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:21:53,935 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19221 total (0 active) Queueing time: mean = 10.058 us, max = 1.423 ms, min = 568.000 ns, total = 193.321 ms Execution time: mean = 27.193 us, total = 522.685 ms Event stats: event_loop_lag_probe - 19195 total (0 active), Execution time: mean = 27.150 us, total = 521.136 ms, Queueing time: mean = 9.974 us, max = 1.423 ms, min = 1.850 us, total = 191.442 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:22:53,935 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 970 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1005 - cumulative published bytes: 799001 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:22:53,935 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 58637 total (4 active) Queueing time: mean = 84.619 us, max = 279.303 ms, min = -0.000 s, total = 4.962 s Execution time: mean = 246.616 us, total = 14.461 s Event stats: event_loop_lag_probe - 19434 total (0 active), Execution time: mean = 24.736 us, total = 480.716 ms, Queueing time: mean = 10.052 us, max = 13.097 ms, min = 1.994 us, total = 195.349 ms GcsInMemoryStore.Put - 7818 total (0 active), Execution time: mean = 41.352 us, total = 323.288 ms, Queueing time: mean = 295.995 us, max = 278.545 ms, min = 2.025 us, total = 2.314 s RayletLoadPulled - 4859 total (1 active), Execution time: mean = 249.488 us, total = 1.212 s, Queueing time: mean = 75.320 us, max = 472.090 us, min = -0.000 s, total = 365.982 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4856 total (0 active), Execution time: mean = 77.487 us, total = 376.279 ms, Queueing time: mean = 104.121 us, max = 769.233 us, min = 11.910 us, total = 505.611 ms NodeManagerService.grpc_client.GetResourceLoad - 4856 total (0 active), Execution time: mean = 1.497 ms, total = 7.268 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1954 total (0 active), Execution time: mean = 42.956 us, total = 83.937 ms, Queueing time: mean = 104.312 us, max = 228.583 us, min = 11.443 us, total = 203.826 ms InternalKVGcsService.grpc_server.InternalKVPut - 1954 total (0 active), Execution time: mean = 569.021 us, total = 1.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1943 total (0 active), Execution time: mean = 35.707 us, total = 69.379 ms, Queueing time: mean = 117.910 us, max = 1.498 ms, min = 20.329 us, total = 229.099 ms NodeInfoGcsService.grpc_server.CheckAlive - 1943 total (0 active), Execution time: mean = 596.082 us, total = 1.158 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1621 total (1 active), Execution time: mean = 9.580 us, total = 15.529 ms, Queueing time: mean = 75.836 us, max = 328.578 us, min = 16.354 us, total = 122.930 ms HealthCheck - 1618 total (0 active), Execution time: mean = 7.757 us, total = 12.551 ms, Queueing time: mean = 102.915 us, max = 399.856 us, min = 16.806 us, total = 166.517 ms GcsInMemoryStore.Get - 1039 total (0 active), Execution time: mean = 21.396 us, total = 22.231 ms, Queueing time: mean = 13.332 us, max = 64.163 us, min = 3.701 us, total = 13.852 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1038 total (0 active), Execution time: mean = 56.798 us, total = 58.956 ms, Queueing time: mean = 106.514 us, max = 269.544 us, min = 13.461 us, total = 110.561 ms InternalKVGcsService.grpc_server.InternalKVGet - 1038 total (0 active), Execution time: mean = 609.979 us, total = 633.158 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 970 total (0 active), Execution time: mean = 92.876 us, total = 90.090 ms, Queueing time: mean = 118.198 us, max = 296.469 us, min = 19.828 us, total = 114.652 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 970 total (0 active), Execution time: mean = 687.080 us, total = 666.467 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 486 total (1 active), Execution time: mean = 1.423 ms, total = 691.507 ms, Queueing time: mean = 71.305 us, max = 315.299 us, min = -0.000 s, total = 34.654 ms GCSServer.deadline_timer.debug_state_event_stats_print - 81 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 126.265 ms, Queueing time: mean = 65.238 us, max = 182.019 us, min = 12.907 us, total = 5.284 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:22:53,935 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 224359 total (1 active) Queueing time: mean = 49.558 us, max = 7.565 ms, min = 1.647 us, total = 11.119 s Execution time: mean = 254.883 us, total = 57.185 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 101976 total (0 active), Execution time: mean = 527.257 us, total = 53.768 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 101976 total (0 active), Execution time: mean = 27.864 us, total = 2.841 s, Queueing time: mean = 106.257 us, max = 7.565 ms, min = 3.372 us, total = 10.836 s event_loop_lag_probe - 19433 total (0 active), Execution time: mean = 29.070 us, total = 564.909 ms, Queueing time: mean = 10.467 us, max = 4.983 ms, min = 1.647 us, total = 203.400 ms GcsTaskManager.GcJobSummary - 973 total (1 active), Execution time: mean = 11.733 us, total = 11.416 ms, Queueing time: mean = 81.745 us, max = 229.756 us, min = 16.725 us, total = 79.538 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:22:53,935 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 23937 total (25 active) Queueing time: mean = 19.623 us, max = 333.684 us, min = 2.330 us, total = 469.719 ms Execution time: mean = 4.787 s, total = 114581.954 s Event stats: event_loop_lag_probe - 19433 total (0 active), Execution time: mean = 31.741 us, total = 616.823 ms, Queueing time: mean = 11.673 us, max = 291.078 us, min = 2.330 us, total = 226.847 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1213 total (0 active), Execution time: mean = 43.383 us, total = 52.624 ms, Queueing time: mean = 104.961 us, max = 333.684 us, min = 6.593 us, total = 127.318 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1213 total (24 active), Execution time: mean = 94.460 s, total = 114580.484 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1005 total (0 active), Execution time: mean = 116.828 us, total = 117.412 ms, Queueing time: mean = 110.228 us, max = 249.252 us, min = 16.251 us, total = 110.779 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1005 total (0 active), Execution time: mean = 652.216 us, total = 655.477 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 648.681 us, total = 11.028 ms, Queueing time: mean = 79.898 us, max = 153.556 us, min = 48.361 us, total = 1.358 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:22:53,935 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19461 total (0 active) Queueing time: mean = 10.052 us, max = 1.423 ms, min = 568.000 ns, total = 195.620 ms Execution time: mean = 27.184 us, total = 529.035 ms Event stats: event_loop_lag_probe - 19435 total (0 active), Execution time: mean = 27.141 us, total = 527.485 ms, Queueing time: mean = 9.969 us, max = 1.423 ms, min = 1.850 us, total = 193.741 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:23:53,936 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 982 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1018 - cumulative published bytes: 808863 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:23:53,937 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 59356 total (4 active) Queueing time: mean = 84.107 us, max = 279.303 ms, min = -0.000 s, total = 4.992 s Execution time: mean = 246.556 us, total = 14.635 s Event stats: event_loop_lag_probe - 19674 total (0 active), Execution time: mean = 24.748 us, total = 486.884 ms, Queueing time: mean = 10.046 us, max = 13.097 ms, min = 1.994 us, total = 197.635 ms GcsInMemoryStore.Put - 7914 total (0 active), Execution time: mean = 40.915 us, total = 323.800 ms, Queueing time: mean = 292.984 us, max = 278.545 ms, min = 2.025 us, total = 2.319 s RayletLoadPulled - 4919 total (1 active), Execution time: mean = 249.594 us, total = 1.228 s, Queueing time: mean = 75.338 us, max = 472.090 us, min = -0.000 s, total = 370.588 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4916 total (0 active), Execution time: mean = 77.481 us, total = 380.896 ms, Queueing time: mean = 104.051 us, max = 769.233 us, min = 11.910 us, total = 511.513 ms NodeManagerService.grpc_client.GetResourceLoad - 4916 total (0 active), Execution time: mean = 1.496 ms, total = 7.355 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 1978 total (0 active), Execution time: mean = 43.068 us, total = 85.189 ms, Queueing time: mean = 104.395 us, max = 228.583 us, min = 11.443 us, total = 206.493 ms InternalKVGcsService.grpc_server.InternalKVPut - 1978 total (0 active), Execution time: mean = 569.042 us, total = 1.126 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1967 total (0 active), Execution time: mean = 35.692 us, total = 70.206 ms, Queueing time: mean = 118.119 us, max = 1.498 ms, min = 20.329 us, total = 232.339 ms NodeInfoGcsService.grpc_server.CheckAlive - 1967 total (0 active), Execution time: mean = 596.041 us, total = 1.172 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1641 total (1 active), Execution time: mean = 9.574 us, total = 15.711 ms, Queueing time: mean = 75.939 us, max = 328.578 us, min = 16.354 us, total = 124.615 ms HealthCheck - 1638 total (0 active), Execution time: mean = 7.758 us, total = 12.707 ms, Queueing time: mean = 102.903 us, max = 399.856 us, min = 16.806 us, total = 168.555 ms GcsInMemoryStore.Get - 1051 total (0 active), Execution time: mean = 21.392 us, total = 22.483 ms, Queueing time: mean = 13.326 us, max = 64.163 us, min = 3.701 us, total = 14.006 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1050 total (0 active), Execution time: mean = 56.938 us, total = 59.785 ms, Queueing time: mean = 106.575 us, max = 269.544 us, min = 13.461 us, total = 111.904 ms InternalKVGcsService.grpc_server.InternalKVGet - 1050 total (0 active), Execution time: mean = 610.017 us, total = 640.518 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 982 total (0 active), Execution time: mean = 92.896 us, total = 91.224 ms, Queueing time: mean = 118.234 us, max = 296.469 us, min = 19.828 us, total = 116.106 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 982 total (0 active), Execution time: mean = 687.368 us, total = 674.995 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 492 total (1 active), Execution time: mean = 1.428 ms, total = 702.476 ms, Queueing time: mean = 71.433 us, max = 315.299 us, min = -0.000 s, total = 35.145 ms GCSServer.deadline_timer.debug_state_event_stats_print - 82 total (1 active, 1 running), Execution time: mean = 1.547 ms, total = 126.884 ms, Queueing time: mean = 64.795 us, max = 182.019 us, min = 12.907 us, total = 5.313 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:23:53,937 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 227131 total (1 active) Queueing time: mean = 49.571 us, max = 7.565 ms, min = 1.647 us, total = 11.259 s Execution time: mean = 254.919 us, total = 57.900 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 103236 total (0 active), Execution time: mean = 527.322 us, total = 54.439 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 103236 total (0 active), Execution time: mean = 27.876 us, total = 2.878 s, Queueing time: mean = 106.286 us, max = 7.565 ms, min = 3.372 us, total = 10.973 s event_loop_lag_probe - 19673 total (0 active), Execution time: mean = 29.072 us, total = 571.927 ms, Queueing time: mean = 10.464 us, max = 4.983 ms, min = 1.647 us, total = 205.856 ms GcsTaskManager.GcJobSummary - 985 total (1 active), Execution time: mean = 11.744 us, total = 11.568 ms, Queueing time: mean = 81.795 us, max = 229.756 us, min = 16.725 us, total = 80.568 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:23:53,937 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24229 total (25 active) Queueing time: mean = 19.631 us, max = 333.684 us, min = 2.330 us, total = 475.633 ms Execution time: mean = 4.732 s, total = 114642.156 s Event stats: event_loop_lag_probe - 19673 total (0 active), Execution time: mean = 31.758 us, total = 624.766 ms, Queueing time: mean = 11.684 us, max = 291.078 us, min = 2.330 us, total = 229.864 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1226 total (0 active), Execution time: mean = 43.400 us, total = 53.208 ms, Queueing time: mean = 105.030 us, max = 333.684 us, min = 6.593 us, total = 128.766 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1226 total (24 active), Execution time: mean = 93.508 s, total = 114640.668 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1018 total (0 active), Execution time: mean = 116.688 us, total = 118.788 ms, Queueing time: mean = 110.243 us, max = 249.252 us, min = 16.251 us, total = 112.228 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1018 total (0 active), Execution time: mean = 651.853 us, total = 663.586 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 648.681 us, total = 11.028 ms, Queueing time: mean = 79.898 us, max = 153.556 us, min = 48.361 us, total = 1.358 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:23:53,937 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19701 total (0 active) Queueing time: mean = 10.044 us, max = 1.423 ms, min = 568.000 ns, total = 197.884 ms Execution time: mean = 27.188 us, total = 535.627 ms Event stats: event_loop_lag_probe - 19675 total (0 active), Execution time: mean = 27.145 us, total = 534.078 ms, Queueing time: mean = 9.962 us, max = 1.423 ms, min = 1.850 us, total = 196.004 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:24:53,938 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 994 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1029 - cumulative published bytes: 817872 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:24:53,939 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 60075 total (4 active) Queueing time: mean = 83.577 us, max = 279.303 ms, min = -0.000 s, total = 5.021 s Execution time: mean = 246.480 us, total = 14.807 s Event stats: event_loop_lag_probe - 19914 total (0 active), Execution time: mean = 24.751 us, total = 492.894 ms, Queueing time: mean = 10.037 us, max = 13.097 ms, min = 1.994 us, total = 199.878 ms GcsInMemoryStore.Put - 8010 total (0 active), Execution time: mean = 40.484 us, total = 324.273 ms, Queueing time: mean = 290.027 us, max = 278.545 ms, min = 2.025 us, total = 2.323 s RayletLoadPulled - 4979 total (1 active), Execution time: mean = 249.428 us, total = 1.242 s, Queueing time: mean = 75.293 us, max = 472.090 us, min = -0.000 s, total = 374.886 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 4976 total (0 active), Execution time: mean = 77.431 us, total = 385.298 ms, Queueing time: mean = 103.850 us, max = 769.233 us, min = 11.910 us, total = 516.758 ms NodeManagerService.grpc_client.GetResourceLoad - 4976 total (0 active), Execution time: mean = 1.496 ms, total = 7.444 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2002 total (0 active), Execution time: mean = 43.230 us, total = 86.547 ms, Queueing time: mean = 104.301 us, max = 228.583 us, min = 11.443 us, total = 208.811 ms InternalKVGcsService.grpc_server.InternalKVPut - 2002 total (0 active), Execution time: mean = 569.398 us, total = 1.140 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 1991 total (0 active), Execution time: mean = 35.702 us, total = 71.082 ms, Queueing time: mean = 118.184 us, max = 1.498 ms, min = 20.329 us, total = 235.305 ms NodeInfoGcsService.grpc_server.CheckAlive - 1991 total (0 active), Execution time: mean = 595.914 us, total = 1.186 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1661 total (1 active), Execution time: mean = 9.569 us, total = 15.894 ms, Queueing time: mean = 76.046 us, max = 328.578 us, min = 16.354 us, total = 126.312 ms HealthCheck - 1658 total (0 active), Execution time: mean = 7.758 us, total = 12.863 ms, Queueing time: mean = 102.925 us, max = 399.856 us, min = 16.806 us, total = 170.650 ms GcsInMemoryStore.Get - 1063 total (0 active), Execution time: mean = 21.375 us, total = 22.721 ms, Queueing time: mean = 13.311 us, max = 64.163 us, min = 3.701 us, total = 14.150 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1062 total (0 active), Execution time: mean = 56.887 us, total = 60.413 ms, Queueing time: mean = 106.608 us, max = 269.544 us, min = 13.461 us, total = 113.217 ms InternalKVGcsService.grpc_server.InternalKVGet - 1062 total (0 active), Execution time: mean = 610.128 us, total = 647.956 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 994 total (0 active), Execution time: mean = 92.793 us, total = 92.236 ms, Queueing time: mean = 118.182 us, max = 296.469 us, min = 19.828 us, total = 117.473 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 994 total (0 active), Execution time: mean = 687.084 us, total = 682.962 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 498 total (1 active), Execution time: mean = 1.427 ms, total = 710.717 ms, Queueing time: mean = 71.470 us, max = 315.299 us, min = -0.000 s, total = 35.592 ms GCSServer.deadline_timer.debug_state_event_stats_print - 83 total (1 active, 1 running), Execution time: mean = 1.550 ms, total = 128.661 ms, Queueing time: mean = 64.795 us, max = 182.019 us, min = 12.907 us, total = 5.378 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:24:53,939 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 229903 total (1 active) Queueing time: mean = 49.607 us, max = 7.565 ms, min = 1.647 us, total = 11.405 s Execution time: mean = 255.040 us, total = 58.635 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 104496 total (0 active), Execution time: mean = 527.582 us, total = 55.130 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 104496 total (0 active), Execution time: mean = 27.880 us, total = 2.913 s, Queueing time: mean = 106.364 us, max = 7.565 ms, min = 3.372 us, total = 11.115 s event_loop_lag_probe - 19913 total (0 active), Execution time: mean = 29.089 us, total = 579.246 ms, Queueing time: mean = 10.465 us, max = 4.983 ms, min = 1.647 us, total = 208.386 ms GcsTaskManager.GcJobSummary - 997 total (1 active), Execution time: mean = 11.737 us, total = 11.702 ms, Queueing time: mean = 81.890 us, max = 229.756 us, min = 16.725 us, total = 81.644 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:24:53,939 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24513 total (25 active) Queueing time: mean = 19.629 us, max = 333.684 us, min = 2.330 us, total = 481.178 ms Execution time: mean = 4.679 s, total = 114697.289 s Event stats: event_loop_lag_probe - 19913 total (0 active), Execution time: mean = 31.781 us, total = 632.857 ms, Queueing time: mean = 11.691 us, max = 291.078 us, min = 2.330 us, total = 232.795 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1237 total (0 active), Execution time: mean = 43.466 us, total = 53.767 ms, Queueing time: mean = 105.153 us, max = 333.684 us, min = 6.593 us, total = 130.074 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1237 total (24 active), Execution time: mean = 92.721 s, total = 114695.782 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1029 total (0 active), Execution time: mean = 117.169 us, total = 120.567 ms, Queueing time: mean = 110.334 us, max = 249.252 us, min = 16.251 us, total = 113.533 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1029 total (0 active), Execution time: mean = 652.684 us, total = 671.612 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 648.681 us, total = 11.028 ms, Queueing time: mean = 79.898 us, max = 153.556 us, min = 48.361 us, total = 1.358 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:24:53,939 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 19941 total (0 active) Queueing time: mean = 10.035 us, max = 1.423 ms, min = 568.000 ns, total = 200.111 ms Execution time: mean = 27.169 us, total = 541.778 ms Event stats: event_loop_lag_probe - 19915 total (0 active), Execution time: mean = 27.127 us, total = 540.229 ms, Queueing time: mean = 9.954 us, max = 1.423 ms, min = 1.850 us, total = 198.232 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:25:53,939 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1005 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1042 - cumulative published bytes: 827734 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:25:53,940 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 60783 total (4 active) Queueing time: mean = 83.141 us, max = 279.303 ms, min = -0.000 s, total = 5.054 s Execution time: mean = 246.499 us, total = 14.983 s Event stats: event_loop_lag_probe - 20154 total (0 active), Execution time: mean = 24.771 us, total = 499.227 ms, Queueing time: mean = 10.110 us, max = 13.097 ms, min = 1.994 us, total = 203.758 ms GcsInMemoryStore.Put - 8104 total (0 active), Execution time: mean = 40.068 us, total = 324.709 ms, Queueing time: mean = 287.225 us, max = 278.545 ms, min = 2.025 us, total = 2.328 s RayletLoadPulled - 5039 total (1 active), Execution time: mean = 249.679 us, total = 1.258 s, Queueing time: mean = 75.380 us, max = 472.090 us, min = -0.000 s, total = 379.841 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5036 total (0 active), Execution time: mean = 77.483 us, total = 390.205 ms, Queueing time: mean = 103.976 us, max = 769.233 us, min = 11.910 us, total = 523.624 ms NodeManagerService.grpc_client.GetResourceLoad - 5036 total (0 active), Execution time: mean = 1.496 ms, total = 7.536 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2024 total (0 active), Execution time: mean = 43.189 us, total = 87.414 ms, Queueing time: mean = 104.273 us, max = 228.583 us, min = 11.443 us, total = 211.049 ms InternalKVGcsService.grpc_server.InternalKVPut - 2024 total (0 active), Execution time: mean = 569.457 us, total = 1.153 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2015 total (0 active), Execution time: mean = 35.720 us, total = 71.975 ms, Queueing time: mean = 118.221 us, max = 1.498 ms, min = 20.329 us, total = 238.216 ms NodeInfoGcsService.grpc_server.CheckAlive - 2015 total (0 active), Execution time: mean = 595.820 us, total = 1.201 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1681 total (1 active), Execution time: mean = 9.564 us, total = 16.077 ms, Queueing time: mean = 76.071 us, max = 328.578 us, min = 16.354 us, total = 127.875 ms HealthCheck - 1678 total (0 active), Execution time: mean = 7.763 us, total = 13.027 ms, Queueing time: mean = 103.220 us, max = 399.856 us, min = 16.806 us, total = 173.203 ms GcsInMemoryStore.Get - 1074 total (0 active), Execution time: mean = 21.347 us, total = 22.927 ms, Queueing time: mean = 13.315 us, max = 64.163 us, min = 3.701 us, total = 14.300 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1073 total (0 active), Execution time: mean = 56.895 us, total = 61.049 ms, Queueing time: mean = 106.657 us, max = 269.544 us, min = 13.461 us, total = 114.443 ms InternalKVGcsService.grpc_server.InternalKVGet - 1073 total (0 active), Execution time: mean = 610.307 us, total = 654.859 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1005 total (0 active), Execution time: mean = 92.896 us, total = 93.361 ms, Queueing time: mean = 118.176 us, max = 296.469 us, min = 19.828 us, total = 118.767 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1005 total (0 active), Execution time: mean = 687.041 us, total = 690.476 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 504 total (1 active), Execution time: mean = 1.429 ms, total = 720.056 ms, Queueing time: mean = 71.371 us, max = 315.299 us, min = -0.000 s, total = 35.971 ms GCSServer.deadline_timer.debug_state_event_stats_print - 84 total (1 active, 1 running), Execution time: mean = 1.553 ms, total = 130.442 ms, Queueing time: mean = 64.996 us, max = 182.019 us, min = 12.907 us, total = 5.460 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:25:53,941 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 232675 total (1 active) Queueing time: mean = 49.641 us, max = 7.565 ms, min = 1.647 us, total = 11.550 s Execution time: mean = 255.185 us, total = 59.375 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 105756 total (0 active), Execution time: mean = 527.886 us, total = 55.827 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 105756 total (0 active), Execution time: mean = 27.891 us, total = 2.950 s, Queueing time: mean = 106.439 us, max = 7.565 ms, min = 3.372 us, total = 11.257 s event_loop_lag_probe - 20153 total (0 active), Execution time: mean = 29.100 us, total = 586.461 ms, Queueing time: mean = 10.468 us, max = 4.983 ms, min = 1.647 us, total = 210.956 ms GcsTaskManager.GcJobSummary - 1009 total (1 active), Execution time: mean = 11.731 us, total = 11.836 ms, Queueing time: mean = 81.974 us, max = 229.756 us, min = 16.725 us, total = 82.712 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:25:53,941 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 24805 total (25 active) Queueing time: mean = 19.628 us, max = 333.684 us, min = 2.330 us, total = 486.861 ms Execution time: mean = 4.626 s, total = 114757.469 s Event stats: event_loop_lag_probe - 20153 total (0 active), Execution time: mean = 31.794 us, total = 640.737 ms, Queueing time: mean = 11.693 us, max = 291.078 us, min = 2.330 us, total = 235.645 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1250 total (0 active), Execution time: mean = 43.509 us, total = 54.387 ms, Queueing time: mean = 105.199 us, max = 333.684 us, min = 6.593 us, total = 131.499 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1250 total (24 active), Execution time: mean = 91.805 s, total = 114755.944 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1042 total (0 active), Execution time: mean = 117.093 us, total = 122.011 ms, Queueing time: mean = 110.308 us, max = 249.252 us, min = 16.251 us, total = 114.941 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1042 total (0 active), Execution time: mean = 652.700 us, total = 680.113 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 17 total (1 active), Execution time: mean = 648.681 us, total = 11.028 ms, Queueing time: mean = 79.898 us, max = 153.556 us, min = 48.361 us, total = 1.358 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:25:53,941 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20180 total (0 active) Queueing time: mean = 10.032 us, max = 1.423 ms, min = 568.000 ns, total = 202.448 ms Execution time: mean = 27.160 us, total = 548.084 ms Event stats: event_loop_lag_probe - 20154 total (0 active), Execution time: mean = 27.118 us, total = 546.535 ms, Queueing time: mean = 9.952 us, max = 1.423 ms, min = 1.850 us, total = 200.569 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:26:53,941 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1017 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1054 - cumulative published bytes: 837562 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:26:53,942 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 61501 total (4 active) Queueing time: mean = 82.673 us, max = 279.303 ms, min = -0.000 s, total = 5.084 s Execution time: mean = 246.541 us, total = 15.163 s Event stats: event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 24.762 us, total = 504.996 ms, Queueing time: mean = 10.101 us, max = 13.097 ms, min = 1.994 us, total = 206.009 ms GcsInMemoryStore.Put - 8200 total (0 active), Execution time: mean = 39.662 us, total = 325.229 ms, Queueing time: mean = 284.418 us, max = 278.545 ms, min = 2.025 us, total = 2.332 s RayletLoadPulled - 5099 total (1 active), Execution time: mean = 249.723 us, total = 1.273 s, Queueing time: mean = 75.396 us, max = 472.090 us, min = -0.000 s, total = 384.446 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5096 total (0 active), Execution time: mean = 77.540 us, total = 395.145 ms, Queueing time: mean = 103.977 us, max = 769.233 us, min = 11.910 us, total = 529.869 ms NodeManagerService.grpc_client.GetResourceLoad - 5096 total (0 active), Execution time: mean = 1.497 ms, total = 7.629 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2048 total (0 active), Execution time: mean = 43.308 us, total = 88.696 ms, Queueing time: mean = 104.410 us, max = 228.583 us, min = 11.443 us, total = 213.831 ms InternalKVGcsService.grpc_server.InternalKVPut - 2048 total (0 active), Execution time: mean = 569.498 us, total = 1.166 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2039 total (0 active), Execution time: mean = 35.723 us, total = 72.839 ms, Queueing time: mean = 118.243 us, max = 1.498 ms, min = 20.329 us, total = 241.097 ms NodeInfoGcsService.grpc_server.CheckAlive - 2039 total (0 active), Execution time: mean = 595.876 us, total = 1.215 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1701 total (1 active), Execution time: mean = 9.558 us, total = 16.258 ms, Queueing time: mean = 76.283 us, max = 391.368 us, min = 16.354 us, total = 129.758 ms HealthCheck - 1697 total (0 active), Execution time: mean = 7.765 us, total = 13.178 ms, Queueing time: mean = 103.357 us, max = 399.856 us, min = 16.806 us, total = 175.396 ms GcsInMemoryStore.Get - 1086 total (0 active), Execution time: mean = 21.345 us, total = 23.180 ms, Queueing time: mean = 13.324 us, max = 64.163 us, min = 3.701 us, total = 14.469 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1085 total (0 active), Execution time: mean = 56.990 us, total = 61.834 ms, Queueing time: mean = 106.789 us, max = 269.544 us, min = 13.461 us, total = 115.866 ms InternalKVGcsService.grpc_server.InternalKVGet - 1085 total (0 active), Execution time: mean = 610.739 us, total = 662.651 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1017 total (0 active), Execution time: mean = 93.117 us, total = 94.700 ms, Queueing time: mean = 118.158 us, max = 296.469 us, min = 19.828 us, total = 120.167 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1017 total (0 active), Execution time: mean = 687.221 us, total = 698.904 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 510 total (1 active), Execution time: mean = 1.429 ms, total = 728.839 ms, Queueing time: mean = 71.436 us, max = 315.299 us, min = -0.000 s, total = 36.432 ms GCSServer.deadline_timer.debug_state_event_stats_print - 85 total (1 active, 1 running), Execution time: mean = 1.554 ms, total = 132.077 ms, Queueing time: mean = 64.970 us, max = 182.019 us, min = 12.907 us, total = 5.522 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:26:53,942 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 235447 total (1 active) Queueing time: mean = 49.646 us, max = 7.565 ms, min = 1.647 us, total = 11.689 s Execution time: mean = 255.277 us, total = 60.104 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 107016 total (0 active), Execution time: mean = 528.065 us, total = 56.511 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 107016 total (0 active), Execution time: mean = 27.910 us, total = 2.987 s, Queueing time: mean = 106.448 us, max = 7.565 ms, min = 3.372 us, total = 11.392 s event_loop_lag_probe - 20393 total (0 active), Execution time: mean = 29.128 us, total = 594.003 ms, Queueing time: mean = 10.476 us, max = 4.983 ms, min = 1.647 us, total = 213.633 ms GcsTaskManager.GcJobSummary - 1021 total (1 active), Execution time: mean = 11.736 us, total = 11.982 ms, Queueing time: mean = 81.938 us, max = 229.756 us, min = 16.725 us, total = 83.659 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:26:53,942 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25098 total (25 active) Queueing time: mean = 19.622 us, max = 333.684 us, min = 2.330 us, total = 492.462 ms Execution time: mean = 4.623 s, total = 116017.680 s Event stats: event_loop_lag_probe - 20393 total (0 active), Execution time: mean = 31.796 us, total = 648.420 ms, Queueing time: mean = 11.695 us, max = 291.078 us, min = 2.330 us, total = 238.500 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1264 total (0 active), Execution time: mean = 43.517 us, total = 55.006 ms, Queueing time: mean = 105.120 us, max = 333.684 us, min = 6.593 us, total = 132.872 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1264 total (24 active), Execution time: mean = 91.785 s, total = 116016.137 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1054 total (0 active), Execution time: mean = 117.142 us, total = 123.468 ms, Queueing time: mean = 110.270 us, max = 249.252 us, min = 16.251 us, total = 116.224 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1054 total (0 active), Execution time: mean = 653.224 us, total = 688.498 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 615.980 us, total = 11.088 ms, Queueing time: mean = 80.481 us, max = 153.556 us, min = 48.361 us, total = 1.449 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:26:53,943 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20420 total (0 active) Queueing time: mean = 10.020 us, max = 1.423 ms, min = 568.000 ns, total = 204.601 ms Execution time: mean = 27.142 us, total = 554.237 ms Event stats: event_loop_lag_probe - 20394 total (0 active), Execution time: mean = 27.101 us, total = 552.688 ms, Queueing time: mean = 9.940 us, max = 1.423 ms, min = 1.850 us, total = 202.722 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:27:53,943 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1029 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1067 - cumulative published bytes: 847424 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:27:53,944 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 62220 total (4 active) Queueing time: mean = 82.209 us, max = 279.303 ms, min = -0.000 s, total = 5.115 s Execution time: mean = 246.534 us, total = 15.339 s Event stats: event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 24.755 us, total = 510.790 ms, Queueing time: mean = 10.090 us, max = 13.097 ms, min = 1.994 us, total = 208.203 ms GcsInMemoryStore.Put - 8296 total (0 active), Execution time: mean = 39.279 us, total = 325.856 ms, Queueing time: mean = 281.675 us, max = 278.545 ms, min = 2.025 us, total = 2.337 s RayletLoadPulled - 5159 total (1 active), Execution time: mean = 249.682 us, total = 1.288 s, Queueing time: mean = 75.434 us, max = 472.090 us, min = -0.000 s, total = 389.164 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5156 total (0 active), Execution time: mean = 77.540 us, total = 399.797 ms, Queueing time: mean = 104.048 us, max = 769.233 us, min = 11.910 us, total = 536.472 ms NodeManagerService.grpc_client.GetResourceLoad - 5156 total (0 active), Execution time: mean = 1.498 ms, total = 7.722 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2072 total (0 active), Execution time: mean = 43.261 us, total = 89.637 ms, Queueing time: mean = 104.408 us, max = 228.583 us, min = 11.443 us, total = 216.333 ms InternalKVGcsService.grpc_server.InternalKVPut - 2072 total (0 active), Execution time: mean = 569.680 us, total = 1.180 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2063 total (0 active), Execution time: mean = 35.717 us, total = 73.685 ms, Queueing time: mean = 118.435 us, max = 1.498 ms, min = 20.329 us, total = 244.331 ms NodeInfoGcsService.grpc_server.CheckAlive - 2063 total (0 active), Execution time: mean = 596.581 us, total = 1.231 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1721 total (1 active), Execution time: mean = 9.557 us, total = 16.448 ms, Queueing time: mean = 76.287 us, max = 391.368 us, min = 16.354 us, total = 131.289 ms HealthCheck - 1717 total (0 active), Execution time: mean = 7.758 us, total = 13.320 ms, Queueing time: mean = 103.368 us, max = 399.856 us, min = 16.806 us, total = 177.483 ms GcsInMemoryStore.Get - 1098 total (0 active), Execution time: mean = 21.332 us, total = 23.423 ms, Queueing time: mean = 13.320 us, max = 64.163 us, min = 3.701 us, total = 14.625 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1097 total (0 active), Execution time: mean = 57.019 us, total = 62.550 ms, Queueing time: mean = 106.747 us, max = 269.544 us, min = 13.461 us, total = 117.101 ms InternalKVGcsService.grpc_server.InternalKVGet - 1097 total (0 active), Execution time: mean = 610.624 us, total = 669.855 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1029 total (0 active), Execution time: mean = 93.071 us, total = 95.770 ms, Queueing time: mean = 118.055 us, max = 296.469 us, min = 19.828 us, total = 121.478 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1029 total (0 active), Execution time: mean = 687.286 us, total = 707.218 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 516 total (1 active), Execution time: mean = 1.427 ms, total = 736.201 ms, Queueing time: mean = 71.275 us, max = 315.299 us, min = -0.000 s, total = 36.778 ms GCSServer.deadline_timer.debug_state_event_stats_print - 86 total (1 active, 1 running), Execution time: mean = 1.555 ms, total = 133.772 ms, Queueing time: mean = 65.198 us, max = 182.019 us, min = 12.907 us, total = 5.607 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:27:53,944 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 238219 total (1 active) Queueing time: mean = 49.667 us, max = 7.565 ms, min = 1.647 us, total = 11.832 s Execution time: mean = 255.388 us, total = 60.838 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 108276 total (0 active), Execution time: mean = 528.298 us, total = 57.202 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 108276 total (0 active), Execution time: mean = 27.921 us, total = 3.023 s, Queueing time: mean = 106.494 us, max = 7.565 ms, min = 3.372 us, total = 11.531 s event_loop_lag_probe - 20633 total (0 active), Execution time: mean = 29.134 us, total = 601.126 ms, Queueing time: mean = 10.479 us, max = 4.983 ms, min = 1.647 us, total = 216.220 ms GcsTaskManager.GcJobSummary - 1033 total (1 active), Execution time: mean = 11.705 us, total = 12.091 ms, Queueing time: mean = 81.859 us, max = 229.756 us, min = 16.725 us, total = 84.561 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:27:53,944 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25390 total (25 active) Queueing time: mean = 19.619 us, max = 333.684 us, min = 2.330 us, total = 498.117 ms Execution time: mean = 4.572 s, total = 116077.796 s Event stats: event_loop_lag_probe - 20633 total (0 active), Execution time: mean = 31.813 us, total = 656.392 ms, Queueing time: mean = 11.705 us, max = 291.078 us, min = 2.330 us, total = 241.504 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1277 total (0 active), Execution time: mean = 43.556 us, total = 55.622 ms, Queueing time: mean = 104.914 us, max = 333.684 us, min = 6.593 us, total = 133.975 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1277 total (24 active), Execution time: mean = 90.898 s, total = 116076.233 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1067 total (0 active), Execution time: mean = 117.481 us, total = 125.353 ms, Queueing time: mean = 110.377 us, max = 249.252 us, min = 16.251 us, total = 117.772 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1067 total (0 active), Execution time: mean = 653.577 us, total = 697.367 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 615.980 us, total = 11.088 ms, Queueing time: mean = 80.481 us, max = 153.556 us, min = 48.361 us, total = 1.449 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:27:53,944 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20660 total (0 active) Queueing time: mean = 10.007 us, max = 1.423 ms, min = 568.000 ns, total = 206.743 ms Execution time: mean = 27.114 us, total = 560.174 ms Event stats: event_loop_lag_probe - 20634 total (0 active), Execution time: mean = 27.073 us, total = 558.625 ms, Queueing time: mean = 9.928 us, max = 1.423 ms, min = 1.850 us, total = 204.864 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:28:53,945 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1041 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1081 - cumulative published bytes: 857320 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:28:53,946 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 62939 total (4 active) Queueing time: mean = 81.720 us, max = 279.303 ms, min = -0.000 s, total = 5.143 s Execution time: mean = 246.516 us, total = 15.515 s Event stats: event_loop_lag_probe - 20874 total (0 active), Execution time: mean = 24.757 us, total = 516.779 ms, Queueing time: mean = 10.081 us, max = 13.097 ms, min = 1.994 us, total = 210.431 ms GcsInMemoryStore.Put - 8392 total (0 active), Execution time: mean = 38.887 us, total = 326.341 ms, Queueing time: mean = 278.957 us, max = 278.545 ms, min = 2.025 us, total = 2.341 s RayletLoadPulled - 5219 total (1 active), Execution time: mean = 249.745 us, total = 1.303 s, Queueing time: mean = 75.419 us, max = 472.090 us, min = -0.000 s, total = 393.612 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5216 total (0 active), Execution time: mean = 77.590 us, total = 404.708 ms, Queueing time: mean = 103.842 us, max = 769.233 us, min = 11.910 us, total = 541.642 ms NodeManagerService.grpc_client.GetResourceLoad - 5216 total (0 active), Execution time: mean = 1.497 ms, total = 7.811 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2096 total (0 active), Execution time: mean = 43.398 us, total = 90.962 ms, Queueing time: mean = 104.483 us, max = 228.583 us, min = 11.443 us, total = 218.996 ms InternalKVGcsService.grpc_server.InternalKVPut - 2096 total (0 active), Execution time: mean = 569.862 us, total = 1.194 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2087 total (0 active), Execution time: mean = 35.847 us, total = 74.813 ms, Queueing time: mean = 118.518 us, max = 1.498 ms, min = 20.329 us, total = 247.347 ms NodeInfoGcsService.grpc_server.CheckAlive - 2087 total (0 active), Execution time: mean = 597.341 us, total = 1.247 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1741 total (1 active), Execution time: mean = 9.563 us, total = 16.649 ms, Queueing time: mean = 76.244 us, max = 391.368 us, min = 16.354 us, total = 132.741 ms HealthCheck - 1737 total (0 active), Execution time: mean = 7.751 us, total = 13.464 ms, Queueing time: mean = 103.293 us, max = 399.856 us, min = 16.806 us, total = 179.420 ms GcsInMemoryStore.Get - 1110 total (0 active), Execution time: mean = 21.320 us, total = 23.665 ms, Queueing time: mean = 13.318 us, max = 64.163 us, min = 3.701 us, total = 14.783 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1109 total (0 active), Execution time: mean = 57.125 us, total = 63.352 ms, Queueing time: mean = 106.611 us, max = 269.544 us, min = 13.461 us, total = 118.231 ms InternalKVGcsService.grpc_server.InternalKVGet - 1109 total (0 active), Execution time: mean = 610.539 us, total = 677.087 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1041 total (0 active), Execution time: mean = 93.122 us, total = 96.940 ms, Queueing time: mean = 118.073 us, max = 296.469 us, min = 19.828 us, total = 122.914 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1041 total (0 active), Execution time: mean = 687.024 us, total = 715.192 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 522 total (1 active), Execution time: mean = 1.427 ms, total = 744.672 ms, Queueing time: mean = 71.216 us, max = 315.299 us, min = -0.000 s, total = 37.174 ms GCSServer.deadline_timer.debug_state_event_stats_print - 87 total (1 active, 1 running), Execution time: mean = 1.557 ms, total = 135.450 ms, Queueing time: mean = 65.151 us, max = 182.019 us, min = 12.907 us, total = 5.668 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:28:53,946 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 240991 total (1 active) Queueing time: mean = 49.698 us, max = 7.565 ms, min = 1.647 us, total = 11.977 s Execution time: mean = 255.534 us, total = 61.581 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 109536 total (0 active), Execution time: mean = 528.596 us, total = 57.900 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 109536 total (0 active), Execution time: mean = 27.937 us, total = 3.060 s, Queueing time: mean = 106.556 us, max = 7.565 ms, min = 3.372 us, total = 11.672 s event_loop_lag_probe - 20873 total (0 active), Execution time: mean = 29.166 us, total = 608.772 ms, Queueing time: mean = 10.488 us, max = 4.983 ms, min = 1.647 us, total = 218.917 ms GcsTaskManager.GcJobSummary - 1045 total (1 active), Execution time: mean = 11.704 us, total = 12.231 ms, Queueing time: mean = 82.289 us, max = 229.756 us, min = 16.725 us, total = 85.992 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:28:53,946 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25686 total (25 active) Queueing time: mean = 19.609 us, max = 333.684 us, min = 2.330 us, total = 503.679 ms Execution time: mean = 4.521 s, total = 116137.985 s Event stats: event_loop_lag_probe - 20873 total (0 active), Execution time: mean = 31.841 us, total = 664.610 ms, Queueing time: mean = 11.713 us, max = 291.078 us, min = 2.330 us, total = 244.482 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1291 total (0 active), Execution time: mean = 43.562 us, total = 56.239 ms, Queueing time: mean = 104.687 us, max = 333.684 us, min = 6.593 us, total = 135.150 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1291 total (24 active), Execution time: mean = 89.958 s, total = 116136.403 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1081 total (0 active), Execution time: mean = 117.331 us, total = 126.835 ms, Queueing time: mean = 110.250 us, max = 249.252 us, min = 16.251 us, total = 119.181 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1081 total (0 active), Execution time: mean = 652.885 us, total = 705.768 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 615.980 us, total = 11.088 ms, Queueing time: mean = 80.481 us, max = 153.556 us, min = 48.361 us, total = 1.449 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:28:53,946 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 20900 total (0 active) Queueing time: mean = 9.997 us, max = 1.423 ms, min = 568.000 ns, total = 208.932 ms Execution time: mean = 27.098 us, total = 566.356 ms Event stats: event_loop_lag_probe - 20874 total (0 active), Execution time: mean = 27.058 us, total = 564.807 ms, Queueing time: mean = 9.919 us, max = 1.423 ms, min = 1.850 us, total = 207.053 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:29:53,946 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1053 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1093 - cumulative published bytes: 867148 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:29:53,947 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 63658 total (4 active) Queueing time: mean = 81.239 us, max = 279.303 ms, min = -0.000 s, total = 5.172 s Execution time: mean = 246.407 us, total = 15.686 s Event stats: event_loop_lag_probe - 21114 total (0 active), Execution time: mean = 24.764 us, total = 522.870 ms, Queueing time: mean = 10.077 us, max = 13.097 ms, min = 1.994 us, total = 212.771 ms GcsInMemoryStore.Put - 8488 total (0 active), Execution time: mean = 38.507 us, total = 326.847 ms, Queueing time: mean = 276.420 us, max = 278.545 ms, min = 2.025 us, total = 2.346 s RayletLoadPulled - 5279 total (1 active), Execution time: mean = 249.747 us, total = 1.318 s, Queueing time: mean = 75.427 us, max = 472.090 us, min = -0.000 s, total = 398.181 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5276 total (0 active), Execution time: mean = 77.583 us, total = 409.327 ms, Queueing time: mean = 103.642 us, max = 769.233 us, min = 11.910 us, total = 546.815 ms NodeManagerService.grpc_client.GetResourceLoad - 5276 total (0 active), Execution time: mean = 1.497 ms, total = 7.899 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2120 total (0 active), Execution time: mean = 43.410 us, total = 92.028 ms, Queueing time: mean = 104.353 us, max = 228.583 us, min = 11.443 us, total = 221.228 ms InternalKVGcsService.grpc_server.InternalKVPut - 2120 total (0 active), Execution time: mean = 569.711 us, total = 1.208 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2111 total (0 active), Execution time: mean = 35.861 us, total = 75.703 ms, Queueing time: mean = 118.345 us, max = 1.498 ms, min = 20.329 us, total = 249.826 ms NodeInfoGcsService.grpc_server.CheckAlive - 2111 total (0 active), Execution time: mean = 596.961 us, total = 1.260 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1761 total (1 active), Execution time: mean = 9.556 us, total = 16.828 ms, Queueing time: mean = 76.094 us, max = 391.368 us, min = 16.354 us, total = 134.001 ms HealthCheck - 1757 total (0 active), Execution time: mean = 7.746 us, total = 13.611 ms, Queueing time: mean = 103.227 us, max = 399.856 us, min = 16.806 us, total = 181.370 ms GcsInMemoryStore.Get - 1122 total (0 active), Execution time: mean = 21.296 us, total = 23.895 ms, Queueing time: mean = 13.308 us, max = 64.163 us, min = 3.701 us, total = 14.932 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1121 total (0 active), Execution time: mean = 57.208 us, total = 64.130 ms, Queueing time: mean = 106.392 us, max = 269.544 us, min = 13.461 us, total = 119.265 ms InternalKVGcsService.grpc_server.InternalKVGet - 1121 total (0 active), Execution time: mean = 610.047 us, total = 683.862 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1053 total (0 active), Execution time: mean = 93.096 us, total = 98.030 ms, Queueing time: mean = 117.887 us, max = 296.469 us, min = 19.828 us, total = 124.136 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1053 total (0 active), Execution time: mean = 686.261 us, total = 722.633 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 528 total (1 active), Execution time: mean = 1.427 ms, total = 753.694 ms, Queueing time: mean = 71.243 us, max = 315.299 us, min = -0.000 s, total = 37.616 ms GCSServer.deadline_timer.debug_state_event_stats_print - 88 total (1 active, 1 running), Execution time: mean = 1.558 ms, total = 137.138 ms, Queueing time: mean = 65.127 us, max = 182.019 us, min = 12.907 us, total = 5.731 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:29:53,948 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 243763 total (1 active) Queueing time: mean = 49.724 us, max = 7.565 ms, min = 1.647 us, total = 12.121 s Execution time: mean = 255.618 us, total = 62.310 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 110796 total (0 active), Execution time: mean = 528.771 us, total = 58.586 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 110796 total (0 active), Execution time: mean = 27.945 us, total = 3.096 s, Queueing time: mean = 106.609 us, max = 7.565 ms, min = 3.372 us, total = 11.812 s event_loop_lag_probe - 21113 total (0 active), Execution time: mean = 29.172 us, total = 615.902 ms, Queueing time: mean = 10.486 us, max = 4.983 ms, min = 1.647 us, total = 221.391 ms GcsTaskManager.GcJobSummary - 1057 total (1 active), Execution time: mean = 11.695 us, total = 12.361 ms, Queueing time: mean = 82.822 us, max = 265.885 us, min = 16.725 us, total = 87.543 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:29:53,948 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 25974 total (25 active) Queueing time: mean = 19.607 us, max = 333.684 us, min = 2.330 us, total = 509.262 ms Execution time: mean = 4.474 s, total = 116198.159 s Event stats: event_loop_lag_probe - 21113 total (0 active), Execution time: mean = 31.843 us, total = 672.296 ms, Queueing time: mean = 11.713 us, max = 291.078 us, min = 2.330 us, total = 247.297 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1303 total (0 active), Execution time: mean = 43.716 us, total = 56.961 ms, Queueing time: mean = 104.700 us, max = 333.684 us, min = 6.593 us, total = 136.424 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1303 total (24 active), Execution time: mean = 89.176 s, total = 116196.560 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1093 total (0 active), Execution time: mean = 117.264 us, total = 128.170 ms, Queueing time: mean = 110.408 us, max = 249.252 us, min = 16.251 us, total = 120.676 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1093 total (0 active), Execution time: mean = 652.959 us, total = 713.684 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 615.980 us, total = 11.088 ms, Queueing time: mean = 80.481 us, max = 153.556 us, min = 48.361 us, total = 1.449 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:29:53,948 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21140 total (0 active) Queueing time: mean = 9.988 us, max = 1.423 ms, min = 568.000 ns, total = 211.151 ms Execution time: mean = 27.079 us, total = 572.443 ms Event stats: event_loop_lag_probe - 21114 total (0 active), Execution time: mean = 27.039 us, total = 570.894 ms, Queueing time: mean = 9.912 us, max = 1.423 ms, min = 1.850 us, total = 209.272 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:30:53,948 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1065 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1105 - cumulative published bytes: 876976 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:30:53,949 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 64376 total (4 active) Queueing time: mean = 80.838 us, max = 279.303 ms, min = -0.000 s, total = 5.204 s Execution time: mean = 246.395 us, total = 15.862 s Event stats: event_loop_lag_probe - 21353 total (0 active), Execution time: mean = 24.800 us, total = 529.555 ms, Queueing time: mean = 10.082 us, max = 13.097 ms, min = 1.994 us, total = 215.283 ms GcsInMemoryStore.Put - 8584 total (0 active), Execution time: mean = 38.139 us, total = 327.388 ms, Queueing time: mean = 273.855 us, max = 278.545 ms, min = 2.025 us, total = 2.351 s RayletLoadPulled - 5339 total (1 active), Execution time: mean = 249.804 us, total = 1.334 s, Queueing time: mean = 75.544 us, max = 472.090 us, min = -0.000 s, total = 403.330 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5336 total (0 active), Execution time: mean = 77.610 us, total = 414.129 ms, Queueing time: mean = 103.714 us, max = 769.233 us, min = 11.910 us, total = 553.415 ms NodeManagerService.grpc_client.GetResourceLoad - 5336 total (0 active), Execution time: mean = 1.497 ms, total = 7.988 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2144 total (0 active), Execution time: mean = 43.366 us, total = 92.977 ms, Queueing time: mean = 104.468 us, max = 228.583 us, min = 11.443 us, total = 223.980 ms InternalKVGcsService.grpc_server.InternalKVPut - 2144 total (0 active), Execution time: mean = 569.778 us, total = 1.222 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2135 total (0 active), Execution time: mean = 35.859 us, total = 76.559 ms, Queueing time: mean = 118.539 us, max = 1.498 ms, min = 20.329 us, total = 253.081 ms NodeInfoGcsService.grpc_server.CheckAlive - 2135 total (0 active), Execution time: mean = 597.360 us, total = 1.275 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1781 total (1 active), Execution time: mean = 9.553 us, total = 17.014 ms, Queueing time: mean = 76.194 us, max = 391.368 us, min = 16.354 us, total = 135.702 ms HealthCheck - 1777 total (0 active), Execution time: mean = 7.750 us, total = 13.772 ms, Queueing time: mean = 103.409 us, max = 399.856 us, min = 16.806 us, total = 183.758 ms GcsInMemoryStore.Get - 1134 total (0 active), Execution time: mean = 21.291 us, total = 24.143 ms, Queueing time: mean = 13.309 us, max = 64.163 us, min = 3.701 us, total = 15.092 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1133 total (0 active), Execution time: mean = 57.224 us, total = 64.834 ms, Queueing time: mean = 106.388 us, max = 269.544 us, min = 13.461 us, total = 120.538 ms InternalKVGcsService.grpc_server.InternalKVGet - 1133 total (0 active), Execution time: mean = 609.611 us, total = 690.689 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1065 total (0 active), Execution time: mean = 93.011 us, total = 99.057 ms, Queueing time: mean = 118.023 us, max = 296.469 us, min = 19.828 us, total = 125.694 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1065 total (0 active), Execution time: mean = 686.305 us, total = 730.915 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 534 total (1 active), Execution time: mean = 1.430 ms, total = 763.459 ms, Queueing time: mean = 71.479 us, max = 315.299 us, min = -0.000 s, total = 38.170 ms GCSServer.deadline_timer.debug_state_event_stats_print - 89 total (1 active, 1 running), Execution time: mean = 1.559 ms, total = 138.761 ms, Queueing time: mean = 65.335 us, max = 182.019 us, min = 12.907 us, total = 5.815 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:30:53,949 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 246534 total (1 active) Queueing time: mean = 49.743 us, max = 7.565 ms, min = 1.647 us, total = 12.263 s Execution time: mean = 255.723 us, total = 63.044 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 112056 total (0 active), Execution time: mean = 528.990 us, total = 59.277 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 112056 total (0 active), Execution time: mean = 27.956 us, total = 3.133 s, Queueing time: mean = 106.650 us, max = 7.565 ms, min = 3.372 us, total = 11.951 s event_loop_lag_probe - 21352 total (0 active), Execution time: mean = 29.160 us, total = 622.634 ms, Queueing time: mean = 10.479 us, max = 4.983 ms, min = 1.647 us, total = 223.748 ms GcsTaskManager.GcJobSummary - 1069 total (1 active), Execution time: mean = 11.682 us, total = 12.488 ms, Queueing time: mean = 83.100 us, max = 265.885 us, min = 16.725 us, total = 88.834 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:30:53,949 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26261 total (25 active) Queueing time: mean = 19.607 us, max = 333.684 us, min = 2.330 us, total = 514.912 ms Execution time: mean = 4.427 s, total = 116258.362 s Event stats: event_loop_lag_probe - 21352 total (0 active), Execution time: mean = 31.838 us, total = 679.808 ms, Queueing time: mean = 11.715 us, max = 291.078 us, min = 2.330 us, total = 250.136 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1315 total (0 active), Execution time: mean = 43.750 us, total = 57.531 ms, Queueing time: mean = 104.771 us, max = 333.684 us, min = 6.593 us, total = 137.773 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1315 total (24 active), Execution time: mean = 88.408 s, total = 116256.746 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1105 total (0 active), Execution time: mean = 117.181 us, total = 129.485 ms, Queueing time: mean = 110.531 us, max = 249.252 us, min = 16.251 us, total = 122.136 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1105 total (0 active), Execution time: mean = 652.885 us, total = 721.438 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 18 total (1 active), Execution time: mean = 615.980 us, total = 11.088 ms, Queueing time: mean = 80.481 us, max = 153.556 us, min = 48.361 us, total = 1.449 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:30:53,950 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21380 total (0 active) Queueing time: mean = 9.994 us, max = 1.423 ms, min = 568.000 ns, total = 213.672 ms Execution time: mean = 27.096 us, total = 579.314 ms Event stats: event_loop_lag_probe - 21354 total (0 active), Execution time: mean = 27.057 us, total = 577.765 ms, Queueing time: mean = 9.918 us, max = 1.423 ms, min = 1.850 us, total = 211.793 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:31:53,950 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1077 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1117 - cumulative published bytes: 886804 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:31:53,951 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 65095 total (4 active) Queueing time: mean = 80.395 us, max = 279.303 ms, min = -0.000 s, total = 5.233 s Execution time: mean = 246.287 us, total = 16.032 s Event stats: event_loop_lag_probe - 21593 total (0 active), Execution time: mean = 24.793 us, total = 535.362 ms, Queueing time: mean = 10.072 us, max = 13.097 ms, min = 1.994 us, total = 217.490 ms GcsInMemoryStore.Put - 8680 total (0 active), Execution time: mean = 37.769 us, total = 327.836 ms, Queueing time: mean = 271.343 us, max = 278.545 ms, min = 2.025 us, total = 2.355 s RayletLoadPulled - 5399 total (1 active), Execution time: mean = 249.670 us, total = 1.348 s, Queueing time: mean = 75.532 us, max = 472.090 us, min = -0.000 s, total = 407.797 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5396 total (0 active), Execution time: mean = 77.579 us, total = 418.618 ms, Queueing time: mean = 103.635 us, max = 769.233 us, min = 11.910 us, total = 559.217 ms NodeManagerService.grpc_client.GetResourceLoad - 5396 total (0 active), Execution time: mean = 1.497 ms, total = 8.076 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2168 total (0 active), Execution time: mean = 43.344 us, total = 93.970 ms, Queueing time: mean = 104.422 us, max = 228.583 us, min = 11.443 us, total = 226.388 ms InternalKVGcsService.grpc_server.InternalKVPut - 2168 total (0 active), Execution time: mean = 569.506 us, total = 1.235 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2159 total (0 active), Execution time: mean = 35.801 us, total = 77.294 ms, Queueing time: mean = 118.585 us, max = 1.498 ms, min = 20.329 us, total = 256.024 ms NodeInfoGcsService.grpc_server.CheckAlive - 2159 total (0 active), Execution time: mean = 597.522 us, total = 1.290 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1801 total (1 active), Execution time: mean = 9.552 us, total = 17.203 ms, Queueing time: mean = 76.168 us, max = 391.368 us, min = 16.354 us, total = 137.178 ms HealthCheck - 1797 total (0 active), Execution time: mean = 7.748 us, total = 13.924 ms, Queueing time: mean = 103.281 us, max = 399.856 us, min = 16.806 us, total = 185.596 ms GcsInMemoryStore.Get - 1146 total (0 active), Execution time: mean = 21.294 us, total = 24.403 ms, Queueing time: mean = 13.360 us, max = 65.040 us, min = 3.701 us, total = 15.311 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1145 total (0 active), Execution time: mean = 57.306 us, total = 65.615 ms, Queueing time: mean = 106.459 us, max = 269.544 us, min = 13.461 us, total = 121.895 ms InternalKVGcsService.grpc_server.InternalKVGet - 1145 total (0 active), Execution time: mean = 609.484 us, total = 697.860 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1077 total (0 active), Execution time: mean = 92.981 us, total = 100.141 ms, Queueing time: mean = 118.197 us, max = 296.469 us, min = 14.122 us, total = 127.298 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1077 total (0 active), Execution time: mean = 686.239 us, total = 739.079 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 540 total (1 active), Execution time: mean = 1.430 ms, total = 771.938 ms, Queueing time: mean = 71.518 us, max = 315.299 us, min = -0.000 s, total = 38.620 ms GCSServer.deadline_timer.debug_state_event_stats_print - 90 total (1 active, 1 running), Execution time: mean = 1.560 ms, total = 140.405 ms, Queueing time: mean = 65.325 us, max = 182.019 us, min = 12.907 us, total = 5.879 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:31:53,951 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 249306 total (1 active) Queueing time: mean = 49.759 us, max = 7.565 ms, min = 1.647 us, total = 12.405 s Execution time: mean = 255.737 us, total = 63.757 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 113316 total (0 active), Execution time: mean = 529.025 us, total = 59.947 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 113316 total (0 active), Execution time: mean = 27.952 us, total = 3.167 s, Queueing time: mean = 106.681 us, max = 7.565 ms, min = 3.372 us, total = 12.089 s event_loop_lag_probe - 21592 total (0 active), Execution time: mean = 29.172 us, total = 629.878 ms, Queueing time: mean = 10.483 us, max = 4.983 ms, min = 1.647 us, total = 226.346 ms GcsTaskManager.GcJobSummary - 1081 total (1 active), Execution time: mean = 11.654 us, total = 12.598 ms, Queueing time: mean = 83.302 us, max = 265.885 us, min = 16.725 us, total = 90.050 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:31:53,952 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26592 total (25 active) Queueing time: mean = 19.655 us, max = 333.684 us, min = 2.330 us, total = 522.665 ms Execution time: mean = 4.848 s, total = 128918.515 s Event stats: event_loop_lag_probe - 21592 total (0 active), Execution time: mean = 31.835 us, total = 687.375 ms, Queueing time: mean = 11.713 us, max = 291.078 us, min = 2.330 us, total = 252.906 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1348 total (0 active), Execution time: mean = 43.386 us, total = 58.485 ms, Queueing time: mean = 104.829 us, max = 333.684 us, min = 6.593 us, total = 141.310 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1348 total (24 active), Execution time: mean = 95.636 s, total = 128916.879 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1117 total (0 active), Execution time: mean = 117.114 us, total = 130.817 ms, Queueing time: mean = 110.574 us, max = 249.252 us, min = 16.251 us, total = 123.511 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1117 total (0 active), Execution time: mean = 652.734 us, total = 729.104 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 682.151 us, total = 12.961 ms, Queueing time: mean = 80.070 us, max = 153.556 us, min = 48.361 us, total = 1.521 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:31:53,952 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21620 total (0 active) Queueing time: mean = 9.984 us, max = 1.423 ms, min = 568.000 ns, total = 215.848 ms Execution time: mean = 27.073 us, total = 585.308 ms Event stats: event_loop_lag_probe - 21594 total (0 active), Execution time: mean = 27.033 us, total = 583.759 ms, Queueing time: mean = 9.909 us, max = 1.423 ms, min = 1.850 us, total = 213.969 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:32:53,952 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1089 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1129 - cumulative published bytes: 896632 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:32:53,953 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 65814 total (4 active) Queueing time: mean = 80.006 us, max = 279.303 ms, min = -0.000 s, total = 5.266 s Execution time: mean = 246.287 us, total = 16.209 s Event stats: event_loop_lag_probe - 21833 total (0 active), Execution time: mean = 24.814 us, total = 541.762 ms, Queueing time: mean = 10.070 us, max = 13.097 ms, min = 1.994 us, total = 219.859 ms GcsInMemoryStore.Put - 8776 total (0 active), Execution time: mean = 37.413 us, total = 328.333 ms, Queueing time: mean = 268.893 us, max = 278.545 ms, min = 2.025 us, total = 2.360 s RayletLoadPulled - 5459 total (1 active), Execution time: mean = 249.699 us, total = 1.363 s, Queueing time: mean = 75.596 us, max = 472.090 us, min = -0.000 s, total = 412.677 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5456 total (0 active), Execution time: mean = 77.591 us, total = 423.335 ms, Queueing time: mean = 103.723 us, max = 769.233 us, min = 11.910 us, total = 565.911 ms NodeManagerService.grpc_client.GetResourceLoad - 5456 total (0 active), Execution time: mean = 1.497 ms, total = 8.165 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2192 total (0 active), Execution time: mean = 43.313 us, total = 94.941 ms, Queueing time: mean = 104.537 us, max = 228.583 us, min = 11.443 us, total = 229.146 ms InternalKVGcsService.grpc_server.InternalKVPut - 2192 total (0 active), Execution time: mean = 569.652 us, total = 1.249 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2183 total (0 active), Execution time: mean = 35.792 us, total = 78.135 ms, Queueing time: mean = 118.733 us, max = 1.498 ms, min = 20.329 us, total = 259.195 ms NodeInfoGcsService.grpc_server.CheckAlive - 2183 total (0 active), Execution time: mean = 597.740 us, total = 1.305 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1821 total (1 active), Execution time: mean = 9.561 us, total = 17.410 ms, Queueing time: mean = 76.253 us, max = 391.368 us, min = 16.354 us, total = 138.856 ms HealthCheck - 1817 total (0 active), Execution time: mean = 7.749 us, total = 14.080 ms, Queueing time: mean = 103.435 us, max = 399.856 us, min = 16.806 us, total = 187.941 ms GcsInMemoryStore.Get - 1158 total (0 active), Execution time: mean = 21.277 us, total = 24.639 ms, Queueing time: mean = 13.369 us, max = 65.040 us, min = 3.701 us, total = 15.481 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1157 total (0 active), Execution time: mean = 57.308 us, total = 66.305 ms, Queueing time: mean = 106.601 us, max = 269.544 us, min = 13.461 us, total = 123.337 ms InternalKVGcsService.grpc_server.InternalKVGet - 1157 total (0 active), Execution time: mean = 609.869 us, total = 705.619 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1089 total (0 active), Execution time: mean = 92.872 us, total = 101.137 ms, Queueing time: mean = 118.398 us, max = 296.469 us, min = 14.122 us, total = 128.935 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1089 total (0 active), Execution time: mean = 686.610 us, total = 747.718 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 546 total (1 active), Execution time: mean = 1.430 ms, total = 780.880 ms, Queueing time: mean = 71.605 us, max = 315.299 us, min = -0.000 s, total = 39.096 ms GCSServer.deadline_timer.debug_state_event_stats_print - 91 total (1 active, 1 running), Execution time: mean = 1.567 ms, total = 142.622 ms, Queueing time: mean = 65.038 us, max = 182.019 us, min = 12.907 us, total = 5.918 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:32:53,953 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 252078 total (1 active) Queueing time: mean = 49.774 us, max = 7.565 ms, min = 1.647 us, total = 12.547 s Execution time: mean = 255.864 us, total = 64.498 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 114576 total (0 active), Execution time: mean = 529.235 us, total = 60.638 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 114576 total (0 active), Execution time: mean = 27.974 us, total = 3.205 s, Queueing time: mean = 106.709 us, max = 7.565 ms, min = 3.372 us, total = 12.226 s event_loop_lag_probe - 21832 total (0 active), Execution time: mean = 29.415 us, total = 642.199 ms, Queueing time: mean = 10.489 us, max = 4.983 ms, min = 1.647 us, total = 229.007 ms GcsTaskManager.GcJobSummary - 1093 total (1 active), Execution time: mean = 11.632 us, total = 12.714 ms, Queueing time: mean = 83.660 us, max = 265.885 us, min = 16.725 us, total = 91.441 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:32:53,953 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 26880 total (25 active) Queueing time: mean = 19.662 us, max = 333.684 us, min = 2.330 us, total = 528.515 ms Execution time: mean = 4.798 s, total = 128978.746 s Event stats: event_loop_lag_probe - 21832 total (0 active), Execution time: mean = 31.850 us, total = 695.341 ms, Queueing time: mean = 11.718 us, max = 291.078 us, min = 2.330 us, total = 255.821 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1360 total (0 active), Execution time: mean = 43.434 us, total = 59.071 ms, Queueing time: mean = 105.012 us, max = 333.684 us, min = 6.593 us, total = 142.816 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1360 total (24 active), Execution time: mean = 94.836 s, total = 128977.092 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1129 total (0 active), Execution time: mean = 117.073 us, total = 132.176 ms, Queueing time: mean = 110.663 us, max = 249.252 us, min = 16.251 us, total = 124.939 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1129 total (0 active), Execution time: mean = 653.077 us, total = 737.324 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 682.151 us, total = 12.961 ms, Queueing time: mean = 80.070 us, max = 153.556 us, min = 48.361 us, total = 1.521 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:32:53,953 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 21860 total (0 active) Queueing time: mean = 9.982 us, max = 1.423 ms, min = 568.000 ns, total = 218.217 ms Execution time: mean = 27.080 us, total = 591.963 ms Event stats: event_loop_lag_probe - 21834 total (0 active), Execution time: mean = 27.041 us, total = 590.414 ms, Queueing time: mean = 9.908 us, max = 1.423 ms, min = 1.850 us, total = 216.338 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:33:53,954 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1101 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1141 - cumulative published bytes: 906460 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:33:53,955 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 66533 total (4 active) Queueing time: mean = 79.594 us, max = 279.303 ms, min = -0.000 s, total = 5.296 s Execution time: mean = 246.305 us, total = 16.387 s Event stats: event_loop_lag_probe - 22073 total (0 active), Execution time: mean = 24.823 us, total = 547.918 ms, Queueing time: mean = 10.064 us, max = 13.097 ms, min = 1.994 us, total = 222.145 ms GcsInMemoryStore.Put - 8872 total (0 active), Execution time: mean = 37.066 us, total = 328.853 ms, Queueing time: mean = 266.555 us, max = 278.545 ms, min = 2.025 us, total = 2.365 s RayletLoadPulled - 5519 total (1 active), Execution time: mean = 249.673 us, total = 1.378 s, Queueing time: mean = 75.570 us, max = 472.090 us, min = -0.000 s, total = 417.072 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5516 total (0 active), Execution time: mean = 77.571 us, total = 427.884 ms, Queueing time: mean = 103.685 us, max = 769.233 us, min = 11.910 us, total = 571.926 ms NodeManagerService.grpc_client.GetResourceLoad - 5516 total (0 active), Execution time: mean = 1.496 ms, total = 8.255 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2216 total (0 active), Execution time: mean = 43.336 us, total = 96.033 ms, Queueing time: mean = 104.562 us, max = 228.583 us, min = 11.443 us, total = 231.709 ms InternalKVGcsService.grpc_server.InternalKVPut - 2216 total (0 active), Execution time: mean = 569.630 us, total = 1.262 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2207 total (0 active), Execution time: mean = 35.894 us, total = 79.219 ms, Queueing time: mean = 118.824 us, max = 1.498 ms, min = 20.329 us, total = 262.245 ms NodeInfoGcsService.grpc_server.CheckAlive - 2207 total (0 active), Execution time: mean = 598.128 us, total = 1.320 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1841 total (1 active), Execution time: mean = 9.562 us, total = 17.604 ms, Queueing time: mean = 76.238 us, max = 391.368 us, min = 16.354 us, total = 140.355 ms HealthCheck - 1837 total (0 active), Execution time: mean = 7.742 us, total = 14.223 ms, Queueing time: mean = 103.507 us, max = 399.856 us, min = 16.806 us, total = 190.142 ms GcsInMemoryStore.Get - 1170 total (0 active), Execution time: mean = 21.268 us, total = 24.884 ms, Queueing time: mean = 13.369 us, max = 65.040 us, min = 3.701 us, total = 15.642 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1169 total (0 active), Execution time: mean = 57.358 us, total = 67.051 ms, Queueing time: mean = 106.394 us, max = 269.544 us, min = 13.461 us, total = 124.374 ms InternalKVGcsService.grpc_server.InternalKVGet - 1169 total (0 active), Execution time: mean = 613.366 us, total = 717.025 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1101 total (0 active), Execution time: mean = 92.883 us, total = 102.264 ms, Queueing time: mean = 118.250 us, max = 296.469 us, min = 14.122 us, total = 130.193 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1101 total (0 active), Execution time: mean = 686.306 us, total = 755.623 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 552 total (1 active), Execution time: mean = 1.430 ms, total = 789.573 ms, Queueing time: mean = 71.705 us, max = 315.299 us, min = -0.000 s, total = 39.581 ms GCSServer.deadline_timer.debug_state_event_stats_print - 92 total (1 active, 1 running), Execution time: mean = 1.568 ms, total = 144.259 ms, Queueing time: mean = 65.049 us, max = 182.019 us, min = 12.907 us, total = 5.984 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:33:53,955 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 254850 total (1 active) Queueing time: mean = 49.796 us, max = 7.565 ms, min = 1.647 us, total = 12.691 s Execution time: mean = 255.959 us, total = 65.231 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 115836 total (0 active), Execution time: mean = 529.433 us, total = 61.327 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 115836 total (0 active), Execution time: mean = 27.988 us, total = 3.242 s, Queueing time: mean = 106.757 us, max = 7.565 ms, min = 3.372 us, total = 12.366 s event_loop_lag_probe - 22072 total (0 active), Execution time: mean = 29.402 us, total = 648.968 ms, Queueing time: mean = 10.485 us, max = 4.983 ms, min = 1.647 us, total = 231.425 ms GcsTaskManager.GcJobSummary - 1105 total (1 active), Execution time: mean = 11.615 us, total = 12.834 ms, Queueing time: mean = 83.853 us, max = 265.885 us, min = 16.725 us, total = 92.658 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:33:53,955 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27168 total (25 active) Queueing time: mean = 19.666 us, max = 333.684 us, min = 2.330 us, total = 534.288 ms Execution time: mean = 4.750 s, total = 129038.919 s Event stats: event_loop_lag_probe - 22072 total (0 active), Execution time: mean = 31.854 us, total = 703.083 ms, Queueing time: mean = 11.722 us, max = 291.078 us, min = 2.330 us, total = 258.736 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1372 total (0 active), Execution time: mean = 43.452 us, total = 59.616 ms, Queueing time: mean = 105.166 us, max = 333.684 us, min = 6.593 us, total = 144.287 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1372 total (24 active), Execution time: mean = 94.050 s, total = 129037.248 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1141 total (0 active), Execution time: mean = 116.938 us, total = 133.426 ms, Queueing time: mean = 110.715 us, max = 249.252 us, min = 16.251 us, total = 126.326 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1141 total (0 active), Execution time: mean = 653.153 us, total = 745.247 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 682.151 us, total = 12.961 ms, Queueing time: mean = 80.070 us, max = 153.556 us, min = 48.361 us, total = 1.521 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:33:53,955 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22100 total (0 active) Queueing time: mean = 9.979 us, max = 1.423 ms, min = 568.000 ns, total = 220.525 ms Execution time: mean = 27.067 us, total = 598.184 ms Event stats: event_loop_lag_probe - 22074 total (0 active), Execution time: mean = 27.029 us, total = 596.635 ms, Queueing time: mean = 9.905 us, max = 1.423 ms, min = 1.850 us, total = 218.646 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:34:53,955 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1113 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1153 - cumulative published bytes: 916288 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:34:53,956 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 67252 total (4 active) Queueing time: mean = 79.223 us, max = 279.303 ms, min = -0.000 s, total = 5.328 s Execution time: mean = 246.359 us, total = 16.568 s Event stats: event_loop_lag_probe - 22313 total (0 active), Execution time: mean = 24.857 us, total = 554.632 ms, Queueing time: mean = 10.068 us, max = 13.097 ms, min = 1.994 us, total = 224.655 ms GcsInMemoryStore.Put - 8968 total (0 active), Execution time: mean = 36.730 us, total = 329.393 ms, Queueing time: mean = 264.199 us, max = 278.545 ms, min = 2.025 us, total = 2.369 s RayletLoadPulled - 5579 total (1 active), Execution time: mean = 249.690 us, total = 1.393 s, Queueing time: mean = 75.584 us, max = 472.090 us, min = -0.000 s, total = 421.681 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5576 total (0 active), Execution time: mean = 77.590 us, total = 432.639 ms, Queueing time: mean = 103.821 us, max = 769.233 us, min = 11.910 us, total = 578.903 ms NodeManagerService.grpc_client.GetResourceLoad - 5576 total (0 active), Execution time: mean = 1.497 ms, total = 8.347 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2240 total (0 active), Execution time: mean = 43.374 us, total = 97.158 ms, Queueing time: mean = 104.670 us, max = 228.583 us, min = 11.443 us, total = 234.461 ms InternalKVGcsService.grpc_server.InternalKVPut - 2240 total (0 active), Execution time: mean = 569.967 us, total = 1.277 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2231 total (0 active), Execution time: mean = 35.912 us, total = 80.120 ms, Queueing time: mean = 119.021 us, max = 1.498 ms, min = 20.329 us, total = 265.535 ms NodeInfoGcsService.grpc_server.CheckAlive - 2231 total (0 active), Execution time: mean = 598.425 us, total = 1.335 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1861 total (1 active), Execution time: mean = 9.561 us, total = 17.794 ms, Queueing time: mean = 76.335 us, max = 391.368 us, min = 16.354 us, total = 142.060 ms HealthCheck - 1857 total (0 active), Execution time: mean = 7.745 us, total = 14.383 ms, Queueing time: mean = 103.657 us, max = 399.856 us, min = 16.806 us, total = 192.492 ms GcsInMemoryStore.Get - 1182 total (0 active), Execution time: mean = 21.254 us, total = 25.123 ms, Queueing time: mean = 13.388 us, max = 65.040 us, min = 3.701 us, total = 15.824 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1181 total (0 active), Execution time: mean = 57.428 us, total = 67.823 ms, Queueing time: mean = 106.426 us, max = 269.544 us, min = 13.461 us, total = 125.689 ms InternalKVGcsService.grpc_server.InternalKVGet - 1181 total (0 active), Execution time: mean = 613.709 us, total = 724.790 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1113 total (0 active), Execution time: mean = 92.921 us, total = 103.421 ms, Queueing time: mean = 118.397 us, max = 296.469 us, min = 14.122 us, total = 131.776 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1113 total (0 active), Execution time: mean = 686.826 us, total = 764.437 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 558 total (1 active), Execution time: mean = 1.431 ms, total = 798.461 ms, Queueing time: mean = 71.822 us, max = 315.299 us, min = -0.000 s, total = 40.077 ms GCSServer.deadline_timer.debug_state_event_stats_print - 93 total (1 active, 1 running), Execution time: mean = 1.568 ms, total = 145.861 ms, Queueing time: mean = 64.971 us, max = 182.019 us, min = 12.907 us, total = 6.042 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:34:53,956 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 257622 total (1 active) Queueing time: mean = 49.828 us, max = 7.565 ms, min = 1.647 us, total = 12.837 s Execution time: mean = 256.099 us, total = 65.977 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 117096 total (0 active), Execution time: mean = 529.722 us, total = 62.028 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 117096 total (0 active), Execution time: mean = 28.006 us, total = 3.279 s, Queueing time: mean = 106.823 us, max = 7.565 ms, min = 3.372 us, total = 12.509 s event_loop_lag_probe - 22312 total (0 active), Execution time: mean = 29.411 us, total = 656.218 ms, Queueing time: mean = 10.487 us, max = 4.983 ms, min = 1.647 us, total = 233.987 ms GcsTaskManager.GcJobSummary - 1117 total (1 active), Execution time: mean = 11.597 us, total = 12.954 ms, Queueing time: mean = 84.179 us, max = 265.885 us, min = 16.725 us, total = 94.028 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:34:53,956 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27456 total (25 active) Queueing time: mean = 19.671 us, max = 333.684 us, min = 2.330 us, total = 540.087 ms Execution time: mean = 4.702 s, total = 129099.026 s Event stats: event_loop_lag_probe - 22312 total (0 active), Execution time: mean = 31.866 us, total = 710.988 ms, Queueing time: mean = 11.733 us, max = 291.078 us, min = 2.330 us, total = 261.793 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1384 total (0 active), Execution time: mean = 43.522 us, total = 60.235 ms, Queueing time: mean = 105.172 us, max = 333.684 us, min = 6.593 us, total = 145.557 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1384 total (24 active), Execution time: mean = 93.278 s, total = 129097.337 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1153 total (0 active), Execution time: mean = 116.920 us, total = 134.809 ms, Queueing time: mean = 110.839 us, max = 249.252 us, min = 16.251 us, total = 127.798 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1153 total (0 active), Execution time: mean = 653.160 us, total = 753.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 682.151 us, total = 12.961 ms, Queueing time: mean = 80.070 us, max = 153.556 us, min = 48.361 us, total = 1.521 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:34:53,956 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22340 total (0 active) Queueing time: mean = 9.987 us, max = 1.423 ms, min = 568.000 ns, total = 223.112 ms Execution time: mean = 27.085 us, total = 605.089 ms Event stats: event_loop_lag_probe - 22314 total (0 active), Execution time: mean = 27.048 us, total = 603.540 ms, Queueing time: mean = 9.915 us, max = 1.423 ms, min = 1.850 us, total = 221.232 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:35:53,956 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1125 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1165 - cumulative published bytes: 926116 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:35:53,957 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 67971 total (4 active) Queueing time: mean = 78.840 us, max = 279.303 ms, min = -0.000 s, total = 5.359 s Execution time: mean = 246.374 us, total = 16.746 s Event stats: event_loop_lag_probe - 22553 total (0 active), Execution time: mean = 24.885 us, total = 561.225 ms, Queueing time: mean = 10.067 us, max = 13.097 ms, min = 1.994 us, total = 227.046 ms GcsInMemoryStore.Put - 9064 total (0 active), Execution time: mean = 36.396 us, total = 329.895 ms, Queueing time: mean = 261.898 us, max = 278.545 ms, min = 2.025 us, total = 2.374 s RayletLoadPulled - 5639 total (1 active), Execution time: mean = 249.648 us, total = 1.408 s, Queueing time: mean = 75.575 us, max = 472.090 us, min = -0.000 s, total = 426.165 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5636 total (0 active), Execution time: mean = 77.625 us, total = 437.494 ms, Queueing time: mean = 103.861 us, max = 769.233 us, min = 11.910 us, total = 585.362 ms NodeManagerService.grpc_client.GetResourceLoad - 5636 total (0 active), Execution time: mean = 1.497 ms, total = 8.439 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2264 total (0 active), Execution time: mean = 43.412 us, total = 98.285 ms, Queueing time: mean = 104.752 us, max = 228.583 us, min = 11.443 us, total = 237.158 ms InternalKVGcsService.grpc_server.InternalKVPut - 2264 total (0 active), Execution time: mean = 570.183 us, total = 1.291 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2255 total (0 active), Execution time: mean = 35.995 us, total = 81.169 ms, Queueing time: mean = 118.955 us, max = 1.498 ms, min = 20.329 us, total = 268.244 ms NodeInfoGcsService.grpc_server.CheckAlive - 2255 total (0 active), Execution time: mean = 598.603 us, total = 1.350 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1881 total (1 active), Execution time: mean = 9.576 us, total = 18.012 ms, Queueing time: mean = 76.431 us, max = 391.368 us, min = 16.354 us, total = 143.767 ms HealthCheck - 1877 total (0 active), Execution time: mean = 7.744 us, total = 14.535 ms, Queueing time: mean = 103.728 us, max = 399.856 us, min = 16.806 us, total = 194.697 ms GcsInMemoryStore.Get - 1194 total (0 active), Execution time: mean = 21.254 us, total = 25.377 ms, Queueing time: mean = 13.456 us, max = 100.625 us, min = 3.701 us, total = 16.067 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1193 total (0 active), Execution time: mean = 57.461 us, total = 68.551 ms, Queueing time: mean = 106.493 us, max = 269.544 us, min = 13.461 us, total = 127.046 ms InternalKVGcsService.grpc_server.InternalKVGet - 1193 total (0 active), Execution time: mean = 613.900 us, total = 732.383 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1125 total (0 active), Execution time: mean = 92.849 us, total = 104.455 ms, Queueing time: mean = 118.527 us, max = 296.469 us, min = 14.122 us, total = 133.342 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1125 total (0 active), Execution time: mean = 686.601 us, total = 772.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 564 total (1 active), Execution time: mean = 1.433 ms, total = 808.141 ms, Queueing time: mean = 72.100 us, max = 315.299 us, min = -0.000 s, total = 40.665 ms GCSServer.deadline_timer.debug_state_event_stats_print - 94 total (1 active, 1 running), Execution time: mean = 1.562 ms, total = 146.842 ms, Queueing time: mean = 64.622 us, max = 182.019 us, min = 12.907 us, total = 6.074 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:35:53,958 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 260390 total (1 active) Queueing time: mean = 49.856 us, max = 7.565 ms, min = 1.647 us, total = 12.982 s Execution time: mean = 256.239 us, total = 66.722 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 118354 total (0 active), Execution time: mean = 530.005 us, total = 62.728 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 118354 total (0 active), Execution time: mean = 28.032 us, total = 3.318 s, Queueing time: mean = 106.886 us, max = 7.565 ms, min = 3.372 us, total = 12.650 s event_loop_lag_probe - 22552 total (0 active), Execution time: mean = 29.408 us, total = 663.205 ms, Queueing time: mean = 10.482 us, max = 4.983 ms, min = 1.647 us, total = 236.398 ms GcsTaskManager.GcJobSummary - 1129 total (1 active), Execution time: mean = 11.596 us, total = 13.092 ms, Queueing time: mean = 84.254 us, max = 265.885 us, min = 16.725 us, total = 95.123 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:35:53,958 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 27744 total (25 active) Queueing time: mean = 19.666 us, max = 333.684 us, min = 2.330 us, total = 545.613 ms Execution time: mean = 4.655 s, total = 129159.193 s Event stats: event_loop_lag_probe - 22552 total (0 active), Execution time: mean = 31.868 us, total = 718.688 ms, Queueing time: mean = 11.739 us, max = 291.078 us, min = 2.330 us, total = 264.747 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1396 total (0 active), Execution time: mean = 43.578 us, total = 60.835 ms, Queueing time: mean = 105.196 us, max = 333.684 us, min = 6.593 us, total = 146.854 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1396 total (24 active), Execution time: mean = 92.520 s, total = 129157.487 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1165 total (0 active), Execution time: mean = 117.063 us, total = 136.378 ms, Queueing time: mean = 110.793 us, max = 249.252 us, min = 16.251 us, total = 129.074 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1165 total (0 active), Execution time: mean = 653.127 us, total = 760.893 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 19 total (1 active), Execution time: mean = 682.151 us, total = 12.961 ms, Queueing time: mean = 80.070 us, max = 153.556 us, min = 48.361 us, total = 1.521 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:35:53,958 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22579 total (0 active) Queueing time: mean = 9.991 us, max = 1.423 ms, min = 568.000 ns, total = 225.581 ms Execution time: mean = 27.099 us, total = 611.871 ms Event stats: event_loop_lag_probe - 22553 total (0 active), Execution time: mean = 27.062 us, total = 610.322 ms, Queueing time: mean = 9.919 us, max = 1.423 ms, min = 1.850 us, total = 223.702 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:36:53,958 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1137 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1177 - cumulative published bytes: 935944 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:36:53,959 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 68690 total (4 active) Queueing time: mean = 78.527 us, max = 279.303 ms, min = -0.000 s, total = 5.394 s Execution time: mean = 246.502 us, total = 16.932 s Event stats: event_loop_lag_probe - 22793 total (0 active), Execution time: mean = 24.925 us, total = 568.125 ms, Queueing time: mean = 10.078 us, max = 13.097 ms, min = 1.994 us, total = 229.700 ms GcsInMemoryStore.Put - 9160 total (0 active), Execution time: mean = 36.072 us, total = 330.423 ms, Queueing time: mean = 259.653 us, max = 278.545 ms, min = 2.025 us, total = 2.378 s RayletLoadPulled - 5699 total (1 active), Execution time: mean = 249.737 us, total = 1.423 s, Queueing time: mean = 75.595 us, max = 472.090 us, min = -0.000 s, total = 430.813 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5696 total (0 active), Execution time: mean = 77.631 us, total = 442.183 ms, Queueing time: mean = 104.468 us, max = 2.744 ms, min = 11.910 us, total = 595.048 ms NodeManagerService.grpc_client.GetResourceLoad - 5696 total (0 active), Execution time: mean = 1.498 ms, total = 8.534 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2288 total (0 active), Execution time: mean = 43.474 us, total = 99.469 ms, Queueing time: mean = 104.897 us, max = 228.583 us, min = 11.443 us, total = 240.004 ms InternalKVGcsService.grpc_server.InternalKVPut - 2288 total (0 active), Execution time: mean = 570.788 us, total = 1.306 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2279 total (0 active), Execution time: mean = 36.036 us, total = 82.127 ms, Queueing time: mean = 119.091 us, max = 1.498 ms, min = 20.329 us, total = 271.408 ms NodeInfoGcsService.grpc_server.CheckAlive - 2279 total (0 active), Execution time: mean = 599.015 us, total = 1.365 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1901 total (1 active), Execution time: mean = 9.583 us, total = 18.218 ms, Queueing time: mean = 76.457 us, max = 391.368 us, min = 16.354 us, total = 145.344 ms HealthCheck - 1897 total (0 active), Execution time: mean = 7.750 us, total = 14.703 ms, Queueing time: mean = 103.838 us, max = 399.856 us, min = 16.806 us, total = 196.981 ms GcsInMemoryStore.Get - 1206 total (0 active), Execution time: mean = 21.240 us, total = 25.616 ms, Queueing time: mean = 13.472 us, max = 100.625 us, min = 3.701 us, total = 16.247 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1205 total (0 active), Execution time: mean = 57.515 us, total = 69.306 ms, Queueing time: mean = 106.662 us, max = 269.544 us, min = 13.461 us, total = 128.527 ms InternalKVGcsService.grpc_server.InternalKVGet - 1205 total (0 active), Execution time: mean = 614.152 us, total = 740.053 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1137 total (0 active), Execution time: mean = 92.863 us, total = 105.585 ms, Queueing time: mean = 118.632 us, max = 296.469 us, min = 14.122 us, total = 134.884 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1137 total (0 active), Execution time: mean = 686.804 us, total = 780.896 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 570 total (1 active), Execution time: mean = 1.435 ms, total = 818.164 ms, Queueing time: mean = 72.161 us, max = 315.299 us, min = -0.000 s, total = 41.132 ms GCSServer.deadline_timer.debug_state_event_stats_print - 95 total (1 active, 1 running), Execution time: mean = 1.565 ms, total = 148.697 ms, Queueing time: mean = 64.492 us, max = 182.019 us, min = 12.907 us, total = 6.127 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:36:53,959 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 263162 total (1 active) Queueing time: mean = 49.872 us, max = 7.565 ms, min = 1.647 us, total = 13.124 s Execution time: mean = 256.350 us, total = 67.462 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 119614 total (0 active), Execution time: mean = 530.222 us, total = 63.422 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 119614 total (0 active), Execution time: mean = 28.060 us, total = 3.356 s, Queueing time: mean = 106.922 us, max = 7.565 ms, min = 3.372 us, total = 12.789 s event_loop_lag_probe - 22792 total (0 active), Execution time: mean = 29.399 us, total = 670.069 ms, Queueing time: mean = 10.477 us, max = 4.983 ms, min = 1.647 us, total = 238.786 ms GcsTaskManager.GcJobSummary - 1141 total (1 active), Execution time: mean = 11.573 us, total = 13.205 ms, Queueing time: mean = 84.199 us, max = 265.885 us, min = 16.725 us, total = 96.071 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:36:53,960 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28043 total (25 active) Queueing time: mean = 19.666 us, max = 333.684 us, min = 2.330 us, total = 551.484 ms Execution time: mean = 4.683 s, total = 131319.389 s Event stats: event_loop_lag_probe - 22792 total (0 active), Execution time: mean = 31.868 us, total = 726.325 ms, Queueing time: mean = 11.737 us, max = 291.078 us, min = 2.330 us, total = 267.511 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1413 total (0 active), Execution time: mean = 43.587 us, total = 61.588 ms, Queueing time: mean = 105.081 us, max = 333.684 us, min = 6.593 us, total = 148.479 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1413 total (24 active), Execution time: mean = 92.935 s, total = 131317.664 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1177 total (0 active), Execution time: mean = 117.242 us, total = 137.994 ms, Queueing time: mean = 110.821 us, max = 249.252 us, min = 16.251 us, total = 130.436 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1177 total (0 active), Execution time: mean = 653.315 us, total = 768.952 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 658.045 us, total = 13.161 ms, Queueing time: mean = 82.031 us, max = 153.556 us, min = 48.361 us, total = 1.641 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:36:53,960 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 22819 total (0 active) Queueing time: mean = 9.999 us, max = 1.423 ms, min = 568.000 ns, total = 228.166 ms Execution time: mean = 27.128 us, total = 619.041 ms Event stats: event_loop_lag_probe - 22793 total (0 active), Execution time: mean = 27.091 us, total = 617.492 ms, Queueing time: mean = 9.928 us, max = 1.423 ms, min = 1.850 us, total = 226.287 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:37:53,960 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1149 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1189 - cumulative published bytes: 945772 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:37:53,961 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 69409 total (4 active) Queueing time: mean = 78.074 us, max = 279.303 ms, min = -0.000 s, total = 5.419 s Execution time: mean = 246.309 us, total = 17.096 s Event stats: event_loop_lag_probe - 23033 total (0 active), Execution time: mean = 24.893 us, total = 573.358 ms, Queueing time: mean = 10.072 us, max = 13.097 ms, min = 1.994 us, total = 231.977 ms GcsInMemoryStore.Put - 9256 total (0 active), Execution time: mean = 35.745 us, total = 330.852 ms, Queueing time: mean = 257.342 us, max = 278.545 ms, min = 2.025 us, total = 2.382 s RayletLoadPulled - 5759 total (1 active), Execution time: mean = 249.455 us, total = 1.437 s, Queueing time: mean = 75.402 us, max = 472.090 us, min = -0.000 s, total = 434.242 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5756 total (0 active), Execution time: mean = 77.565 us, total = 446.465 ms, Queueing time: mean = 104.271 us, max = 2.744 ms, min = 11.910 us, total = 600.184 ms NodeManagerService.grpc_client.GetResourceLoad - 5756 total (0 active), Execution time: mean = 1.497 ms, total = 8.619 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2312 total (0 active), Execution time: mean = 43.523 us, total = 100.625 ms, Queueing time: mean = 104.734 us, max = 228.583 us, min = 11.443 us, total = 242.144 ms InternalKVGcsService.grpc_server.InternalKVPut - 2312 total (0 active), Execution time: mean = 570.225 us, total = 1.318 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2303 total (0 active), Execution time: mean = 36.058 us, total = 83.041 ms, Queueing time: mean = 118.921 us, max = 1.498 ms, min = 20.329 us, total = 273.874 ms NodeInfoGcsService.grpc_server.CheckAlive - 2303 total (0 active), Execution time: mean = 598.887 us, total = 1.379 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1921 total (1 active), Execution time: mean = 9.564 us, total = 18.373 ms, Queueing time: mean = 76.384 us, max = 391.368 us, min = 11.350 us, total = 146.733 ms HealthCheck - 1917 total (0 active), Execution time: mean = 7.739 us, total = 14.836 ms, Queueing time: mean = 103.700 us, max = 399.856 us, min = 16.806 us, total = 198.793 ms GcsInMemoryStore.Get - 1218 total (0 active), Execution time: mean = 21.196 us, total = 25.817 ms, Queueing time: mean = 13.472 us, max = 100.625 us, min = 3.701 us, total = 16.408 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1217 total (0 active), Execution time: mean = 57.658 us, total = 70.170 ms, Queueing time: mean = 106.405 us, max = 269.544 us, min = 13.461 us, total = 129.495 ms InternalKVGcsService.grpc_server.InternalKVGet - 1217 total (0 active), Execution time: mean = 613.783 us, total = 746.974 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1149 total (0 active), Execution time: mean = 93.061 us, total = 106.927 ms, Queueing time: mean = 118.515 us, max = 296.469 us, min = 14.122 us, total = 136.173 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1149 total (0 active), Execution time: mean = 686.854 us, total = 789.195 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 576 total (1 active), Execution time: mean = 1.433 ms, total = 825.688 ms, Queueing time: mean = 72.030 us, max = 315.299 us, min = -0.000 s, total = 41.489 ms GCSServer.deadline_timer.debug_state_event_stats_print - 96 total (1 active, 1 running), Execution time: mean = 1.567 ms, total = 150.395 ms, Queueing time: mean = 64.445 us, max = 182.019 us, min = 12.907 us, total = 6.187 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:37:53,961 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 265934 total (1 active) Queueing time: mean = 49.877 us, max = 7.565 ms, min = 1.647 us, total = 13.264 s Execution time: mean = 256.323 us, total = 68.165 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 120874 total (0 active), Execution time: mean = 530.174 us, total = 64.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 120874 total (0 active), Execution time: mean = 28.056 us, total = 3.391 s, Queueing time: mean = 106.936 us, max = 7.565 ms, min = 3.372 us, total = 12.926 s event_loop_lag_probe - 23032 total (0 active), Execution time: mean = 29.363 us, total = 676.299 ms, Queueing time: mean = 10.461 us, max = 4.983 ms, min = 1.647 us, total = 240.933 ms GcsTaskManager.GcJobSummary - 1153 total (1 active), Execution time: mean = 11.584 us, total = 13.356 ms, Queueing time: mean = 84.212 us, max = 265.885 us, min = 16.725 us, total = 97.096 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:37:53,962 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28331 total (25 active) Queueing time: mean = 19.648 us, max = 333.684 us, min = 2.330 us, total = 556.638 ms Execution time: mean = 4.637 s, total = 131379.620 s Event stats: event_loop_lag_probe - 23032 total (0 active), Execution time: mean = 31.830 us, total = 733.113 ms, Queueing time: mean = 11.724 us, max = 291.078 us, min = 2.330 us, total = 270.034 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1425 total (0 active), Execution time: mean = 43.635 us, total = 62.179 ms, Queueing time: mean = 105.138 us, max = 333.684 us, min = 6.593 us, total = 149.822 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1425 total (24 active), Execution time: mean = 92.195 s, total = 131377.879 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1189 total (0 active), Execution time: mean = 117.160 us, total = 139.304 ms, Queueing time: mean = 110.785 us, max = 249.252 us, min = 16.251 us, total = 131.724 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1189 total (0 active), Execution time: mean = 653.190 us, total = 776.642 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 658.045 us, total = 13.161 ms, Queueing time: mean = 82.031 us, max = 153.556 us, min = 48.361 us, total = 1.641 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:37:53,962 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23059 total (0 active) Queueing time: mean = 9.979 us, max = 1.423 ms, min = 568.000 ns, total = 230.106 ms Execution time: mean = 27.077 us, total = 624.371 ms Event stats: event_loop_lag_probe - 23033 total (0 active), Execution time: mean = 27.040 us, total = 622.822 ms, Queueing time: mean = 9.909 us, max = 1.423 ms, min = 1.850 us, total = 228.227 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:38:53,962 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1161 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1201 - cumulative published bytes: 955600 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:38:53,963 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 70128 total (4 active) Queueing time: mean = 77.685 us, max = 279.303 ms, min = -0.000 s, total = 5.448 s Execution time: mean = 246.295 us, total = 17.272 s Event stats: event_loop_lag_probe - 23273 total (0 active), Execution time: mean = 24.896 us, total = 579.403 ms, Queueing time: mean = 10.062 us, max = 13.097 ms, min = 1.994 us, total = 234.172 ms GcsInMemoryStore.Put - 9352 total (0 active), Execution time: mean = 35.428 us, total = 331.326 ms, Queueing time: mean = 255.164 us, max = 278.545 ms, min = 2.025 us, total = 2.386 s RayletLoadPulled - 5819 total (1 active), Execution time: mean = 249.631 us, total = 1.453 s, Queueing time: mean = 75.473 us, max = 472.090 us, min = -0.000 s, total = 439.179 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5816 total (0 active), Execution time: mean = 77.572 us, total = 451.158 ms, Queueing time: mean = 104.275 us, max = 2.744 ms, min = 11.910 us, total = 606.463 ms NodeManagerService.grpc_client.GetResourceLoad - 5816 total (0 active), Execution time: mean = 1.498 ms, total = 8.710 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2336 total (0 active), Execution time: mean = 43.441 us, total = 101.478 ms, Queueing time: mean = 104.511 us, max = 228.583 us, min = 11.443 us, total = 244.137 ms InternalKVGcsService.grpc_server.InternalKVPut - 2336 total (0 active), Execution time: mean = 570.140 us, total = 1.332 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2327 total (0 active), Execution time: mean = 36.075 us, total = 83.947 ms, Queueing time: mean = 119.086 us, max = 1.498 ms, min = 20.329 us, total = 277.114 ms NodeInfoGcsService.grpc_server.CheckAlive - 2327 total (0 active), Execution time: mean = 599.252 us, total = 1.394 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1941 total (1 active), Execution time: mean = 9.557 us, total = 18.551 ms, Queueing time: mean = 76.297 us, max = 391.368 us, min = 11.350 us, total = 148.092 ms HealthCheck - 1937 total (0 active), Execution time: mean = 7.755 us, total = 15.021 ms, Queueing time: mean = 103.597 us, max = 399.856 us, min = 16.806 us, total = 200.667 ms GcsInMemoryStore.Get - 1230 total (0 active), Execution time: mean = 21.170 us, total = 26.039 ms, Queueing time: mean = 13.475 us, max = 100.625 us, min = 3.701 us, total = 16.574 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1229 total (0 active), Execution time: mean = 57.691 us, total = 70.902 ms, Queueing time: mean = 106.122 us, max = 269.544 us, min = 13.461 us, total = 130.424 ms InternalKVGcsService.grpc_server.InternalKVGet - 1229 total (0 active), Execution time: mean = 614.005 us, total = 754.612 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1161 total (0 active), Execution time: mean = 93.093 us, total = 108.081 ms, Queueing time: mean = 118.227 us, max = 296.469 us, min = 14.122 us, total = 137.262 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1161 total (0 active), Execution time: mean = 686.758 us, total = 797.326 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 582 total (1 active), Execution time: mean = 1.431 ms, total = 833.016 ms, Queueing time: mean = 71.963 us, max = 315.299 us, min = -0.000 s, total = 41.883 ms GCSServer.deadline_timer.debug_state_event_stats_print - 97 total (1 active, 1 running), Execution time: mean = 1.569 ms, total = 152.169 ms, Queueing time: mean = 64.475 us, max = 182.019 us, min = 12.907 us, total = 6.254 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:38:53,963 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 268706 total (1 active) Queueing time: mean = 49.903 us, max = 7.565 ms, min = 1.647 us, total = 13.409 s Execution time: mean = 256.458 us, total = 68.912 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 122134 total (0 active), Execution time: mean = 530.444 us, total = 64.785 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 122134 total (0 active), Execution time: mean = 28.085 us, total = 3.430 s, Queueing time: mean = 106.996 us, max = 7.565 ms, min = 3.372 us, total = 13.068 s event_loop_lag_probe - 23272 total (0 active), Execution time: mean = 29.344 us, total = 682.898 ms, Queueing time: mean = 10.453 us, max = 4.983 ms, min = 1.647 us, total = 243.268 ms GcsTaskManager.GcJobSummary - 1165 total (1 active), Execution time: mean = 11.601 us, total = 13.515 ms, Queueing time: mean = 84.185 us, max = 265.885 us, min = 16.725 us, total = 98.076 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:38:53,963 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28619 total (25 active) Queueing time: mean = 19.640 us, max = 333.684 us, min = 2.330 us, total = 562.076 ms Execution time: mean = 4.593 s, total = 131439.771 s Event stats: event_loop_lag_probe - 23272 total (0 active), Execution time: mean = 31.831 us, total = 740.780 ms, Queueing time: mean = 11.724 us, max = 291.078 us, min = 2.330 us, total = 272.844 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1437 total (0 active), Execution time: mean = 43.733 us, total = 62.844 ms, Queueing time: mean = 105.109 us, max = 333.684 us, min = 6.593 us, total = 151.042 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1437 total (24 active), Execution time: mean = 91.467 s, total = 131438.011 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1201 total (0 active), Execution time: mean = 117.165 us, total = 140.715 ms, Queueing time: mean = 110.851 us, max = 249.252 us, min = 16.251 us, total = 133.132 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1201 total (0 active), Execution time: mean = 653.569 us, total = 784.936 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 658.045 us, total = 13.161 ms, Queueing time: mean = 82.031 us, max = 153.556 us, min = 48.361 us, total = 1.641 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:38:53,964 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23299 total (0 active) Queueing time: mean = 9.974 us, max = 1.423 ms, min = 568.000 ns, total = 232.393 ms Execution time: mean = 27.068 us, total = 630.655 ms Event stats: event_loop_lag_probe - 23273 total (0 active), Execution time: mean = 27.032 us, total = 629.106 ms, Queueing time: mean = 9.905 us, max = 1.423 ms, min = 1.850 us, total = 230.514 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:39:53,964 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1173 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1213 - cumulative published bytes: 965428 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:39:53,965 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 70847 total (4 active) Queueing time: mean = 77.273 us, max = 279.303 ms, min = -0.000 s, total = 5.475 s Execution time: mean = 246.234 us, total = 17.445 s Event stats: event_loop_lag_probe - 23513 total (0 active), Execution time: mean = 24.867 us, total = 584.694 ms, Queueing time: mean = 10.042 us, max = 13.097 ms, min = 1.994 us, total = 236.126 ms GcsInMemoryStore.Put - 9448 total (0 active), Execution time: mean = 35.116 us, total = 331.773 ms, Queueing time: mean = 253.017 us, max = 278.545 ms, min = 2.025 us, total = 2.391 s RayletLoadPulled - 5879 total (1 active), Execution time: mean = 249.520 us, total = 1.467 s, Queueing time: mean = 75.376 us, max = 472.090 us, min = -0.000 s, total = 443.134 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5876 total (0 active), Execution time: mean = 77.498 us, total = 455.380 ms, Queueing time: mean = 104.058 us, max = 2.744 ms, min = 11.910 us, total = 611.447 ms NodeManagerService.grpc_client.GetResourceLoad - 5876 total (0 active), Execution time: mean = 1.498 ms, total = 8.802 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2360 total (0 active), Execution time: mean = 43.494 us, total = 102.646 ms, Queueing time: mean = 104.495 us, max = 228.583 us, min = 11.443 us, total = 246.609 ms InternalKVGcsService.grpc_server.InternalKVPut - 2360 total (0 active), Execution time: mean = 570.131 us, total = 1.346 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2351 total (0 active), Execution time: mean = 36.039 us, total = 84.728 ms, Queueing time: mean = 119.029 us, max = 1.498 ms, min = 20.329 us, total = 279.838 ms NodeInfoGcsService.grpc_server.CheckAlive - 2351 total (0 active), Execution time: mean = 598.959 us, total = 1.408 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1961 total (1 active), Execution time: mean = 9.537 us, total = 18.703 ms, Queueing time: mean = 76.103 us, max = 391.368 us, min = 11.350 us, total = 149.238 ms HealthCheck - 1957 total (0 active), Execution time: mean = 7.746 us, total = 15.158 ms, Queueing time: mean = 103.416 us, max = 399.856 us, min = 16.806 us, total = 202.386 ms GcsInMemoryStore.Get - 1242 total (0 active), Execution time: mean = 21.162 us, total = 26.284 ms, Queueing time: mean = 13.545 us, max = 128.508 us, min = 3.701 us, total = 16.823 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1241 total (0 active), Execution time: mean = 57.652 us, total = 71.546 ms, Queueing time: mean = 106.006 us, max = 269.544 us, min = 13.461 us, total = 131.554 ms InternalKVGcsService.grpc_server.InternalKVGet - 1241 total (0 active), Execution time: mean = 613.820 us, total = 761.751 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1173 total (0 active), Execution time: mean = 93.066 us, total = 109.166 ms, Queueing time: mean = 118.144 us, max = 296.469 us, min = 14.122 us, total = 138.583 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1173 total (0 active), Execution time: mean = 686.819 us, total = 805.639 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 588 total (1 active), Execution time: mean = 1.430 ms, total = 840.834 ms, Queueing time: mean = 72.520 us, max = 323.875 us, min = -0.000 s, total = 42.642 ms GCSServer.deadline_timer.debug_state_event_stats_print - 98 total (1 active, 1 running), Execution time: mean = 1.572 ms, total = 154.036 ms, Queueing time: mean = 64.363 us, max = 182.019 us, min = 12.907 us, total = 6.308 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:39:53,965 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 271477 total (1 active) Queueing time: mean = 49.926 us, max = 7.565 ms, min = 1.647 us, total = 13.554 s Execution time: mean = 256.551 us, total = 69.648 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 123394 total (0 active), Execution time: mean = 530.617 us, total = 65.475 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 123394 total (0 active), Execution time: mean = 28.115 us, total = 3.469 s, Queueing time: mean = 107.045 us, max = 7.565 ms, min = 3.372 us, total = 13.209 s event_loop_lag_probe - 23511 total (0 active), Execution time: mean = 29.341 us, total = 689.844 ms, Queueing time: mean = 10.449 us, max = 4.983 ms, min = 1.647 us, total = 245.671 ms GcsTaskManager.GcJobSummary - 1177 total (1 active), Execution time: mean = 11.626 us, total = 13.684 ms, Queueing time: mean = 84.214 us, max = 265.885 us, min = 16.725 us, total = 99.120 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:39:53,965 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 28906 total (25 active) Queueing time: mean = 19.638 us, max = 333.684 us, min = 2.330 us, total = 567.667 ms Execution time: mean = 4.549 s, total = 131499.976 s Event stats: event_loop_lag_probe - 23511 total (0 active), Execution time: mean = 31.844 us, total = 748.679 ms, Queueing time: mean = 11.731 us, max = 291.078 us, min = 2.330 us, total = 275.800 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1449 total (0 active), Execution time: mean = 43.773 us, total = 63.427 ms, Queueing time: mean = 105.057 us, max = 333.684 us, min = 6.593 us, total = 152.227 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1449 total (24 active), Execution time: mean = 90.751 s, total = 131498.199 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1213 total (0 active), Execution time: mean = 117.178 us, total = 142.137 ms, Queueing time: mean = 110.949 us, max = 249.252 us, min = 16.251 us, total = 134.581 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1213 total (0 active), Execution time: mean = 653.761 us, total = 793.013 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 658.045 us, total = 13.161 ms, Queueing time: mean = 82.031 us, max = 153.556 us, min = 48.361 us, total = 1.641 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:39:53,966 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23539 total (0 active) Queueing time: mean = 9.960 us, max = 1.423 ms, min = 568.000 ns, total = 234.449 ms Execution time: mean = 27.034 us, total = 636.358 ms Event stats: event_loop_lag_probe - 23513 total (0 active), Execution time: mean = 26.998 us, total = 634.809 ms, Queueing time: mean = 9.891 us, max = 1.423 ms, min = 1.850 us, total = 232.570 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:40:53,966 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1185 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1226 - cumulative published bytes: 975290 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:40:53,967 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 71565 total (4 active) Queueing time: mean = 76.891 us, max = 279.303 ms, min = -0.000 s, total = 5.503 s Execution time: mean = 246.185 us, total = 17.618 s Event stats: event_loop_lag_probe - 23752 total (0 active), Execution time: mean = 24.872 us, total = 590.772 ms, Queueing time: mean = 10.037 us, max = 13.097 ms, min = 1.994 us, total = 238.393 ms GcsInMemoryStore.Put - 9544 total (0 active), Execution time: mean = 34.809 us, total = 332.216 ms, Queueing time: mean = 250.915 us, max = 278.545 ms, min = 2.025 us, total = 2.395 s RayletLoadPulled - 5939 total (1 active), Execution time: mean = 249.516 us, total = 1.482 s, Queueing time: mean = 75.349 us, max = 472.090 us, min = -0.000 s, total = 447.498 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5936 total (0 active), Execution time: mean = 77.487 us, total = 459.960 ms, Queueing time: mean = 104.039 us, max = 2.744 ms, min = 11.910 us, total = 617.573 ms NodeManagerService.grpc_client.GetResourceLoad - 5936 total (0 active), Execution time: mean = 1.498 ms, total = 8.892 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2384 total (0 active), Execution time: mean = 43.428 us, total = 103.532 ms, Queueing time: mean = 104.320 us, max = 228.583 us, min = 11.443 us, total = 248.698 ms InternalKVGcsService.grpc_server.InternalKVPut - 2384 total (0 active), Execution time: mean = 570.112 us, total = 1.359 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2375 total (0 active), Execution time: mean = 35.977 us, total = 85.445 ms, Queueing time: mean = 118.914 us, max = 1.498 ms, min = 20.329 us, total = 282.421 ms NodeInfoGcsService.grpc_server.CheckAlive - 2375 total (0 active), Execution time: mean = 598.866 us, total = 1.422 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 1981 total (1 active), Execution time: mean = 9.549 us, total = 18.917 ms, Queueing time: mean = 76.085 us, max = 391.368 us, min = 11.350 us, total = 150.724 ms HealthCheck - 1977 total (0 active), Execution time: mean = 7.747 us, total = 15.316 ms, Queueing time: mean = 103.425 us, max = 399.856 us, min = 16.806 us, total = 204.471 ms GcsInMemoryStore.Get - 1254 total (0 active), Execution time: mean = 21.122 us, total = 26.486 ms, Queueing time: mean = 13.541 us, max = 128.508 us, min = 3.701 us, total = 16.980 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1253 total (0 active), Execution time: mean = 57.663 us, total = 72.251 ms, Queueing time: mean = 105.763 us, max = 269.544 us, min = 13.461 us, total = 132.521 ms InternalKVGcsService.grpc_server.InternalKVGet - 1253 total (0 active), Execution time: mean = 613.508 us, total = 768.725 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1185 total (0 active), Execution time: mean = 92.973 us, total = 110.173 ms, Queueing time: mean = 117.914 us, max = 296.469 us, min = 14.122 us, total = 139.728 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1185 total (0 active), Execution time: mean = 686.475 us, total = 813.473 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 594 total (1 active), Execution time: mean = 1.430 ms, total = 849.453 ms, Queueing time: mean = 72.710 us, max = 323.875 us, min = -0.000 s, total = 43.190 ms GCSServer.deadline_timer.debug_state_event_stats_print - 99 total (1 active, 1 running), Execution time: mean = 1.575 ms, total = 155.970 ms, Queueing time: mean = 64.520 us, max = 182.019 us, min = 12.907 us, total = 6.387 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:40:53,967 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 274249 total (1 active) Queueing time: mean = 49.914 us, max = 7.565 ms, min = 1.647 us, total = 13.689 s Execution time: mean = 256.561 us, total = 70.362 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 124654 total (0 active), Execution time: mean = 530.630 us, total = 66.145 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 124654 total (0 active), Execution time: mean = 28.124 us, total = 3.506 s, Queueing time: mean = 107.018 us, max = 7.565 ms, min = 3.372 us, total = 13.340 s event_loop_lag_probe - 23751 total (0 active), Execution time: mean = 29.345 us, total = 696.980 ms, Queueing time: mean = 10.451 us, max = 4.983 ms, min = 1.647 us, total = 248.228 ms GcsTaskManager.GcJobSummary - 1189 total (1 active), Execution time: mean = 11.629 us, total = 13.827 ms, Queueing time: mean = 84.268 us, max = 265.885 us, min = 16.725 us, total = 100.195 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:40:53,967 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29198 total (25 active) Queueing time: mean = 19.646 us, max = 333.684 us, min = 2.330 us, total = 573.631 ms Execution time: mean = 4.506 s, total = 131560.201 s Event stats: event_loop_lag_probe - 23751 total (0 active), Execution time: mean = 31.833 us, total = 756.070 ms, Queueing time: mean = 11.729 us, max = 291.078 us, min = 2.330 us, total = 278.573 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1462 total (0 active), Execution time: mean = 43.836 us, total = 64.089 ms, Queueing time: mean = 105.132 us, max = 333.684 us, min = 6.593 us, total = 153.703 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1462 total (24 active), Execution time: mean = 89.985 s, total = 131558.405 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1226 total (0 active), Execution time: mean = 117.094 us, total = 143.558 ms, Queueing time: mean = 111.173 us, max = 249.252 us, min = 16.251 us, total = 136.298 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1226 total (0 active), Execution time: mean = 654.055 us, total = 801.872 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 20 total (1 active), Execution time: mean = 658.045 us, total = 13.161 ms, Queueing time: mean = 82.031 us, max = 153.556 us, min = 48.361 us, total = 1.641 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:40:53,967 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 23779 total (0 active) Queueing time: mean = 9.959 us, max = 1.423 ms, min = 568.000 ns, total = 236.817 ms Execution time: mean = 27.033 us, total = 642.829 ms Event stats: event_loop_lag_probe - 23753 total (0 active), Execution time: mean = 26.998 us, total = 641.279 ms, Queueing time: mean = 9.891 us, max = 1.423 ms, min = 1.850 us, total = 234.938 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:41:53,967 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1197 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1238 - cumulative published bytes: 985118 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:41:53,968 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 72284 total (4 active) Queueing time: mean = 76.528 us, max = 279.303 ms, min = -0.000 s, total = 5.532 s Execution time: mean = 246.067 us, total = 17.787 s Event stats: event_loop_lag_probe - 23992 total (0 active), Execution time: mean = 24.883 us, total = 597.001 ms, Queueing time: mean = 10.033 us, max = 13.097 ms, min = 1.994 us, total = 240.720 ms GcsInMemoryStore.Put - 9640 total (0 active), Execution time: mean = 34.512 us, total = 332.691 ms, Queueing time: mean = 248.887 us, max = 278.545 ms, min = 2.025 us, total = 2.399 s RayletLoadPulled - 5999 total (1 active), Execution time: mean = 249.480 us, total = 1.497 s, Queueing time: mean = 75.336 us, max = 472.090 us, min = -0.000 s, total = 451.939 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 5996 total (0 active), Execution time: mean = 77.467 us, total = 464.494 ms, Queueing time: mean = 103.995 us, max = 2.744 ms, min = 11.910 us, total = 623.552 ms NodeManagerService.grpc_client.GetResourceLoad - 5996 total (0 active), Execution time: mean = 1.497 ms, total = 8.979 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2408 total (0 active), Execution time: mean = 43.457 us, total = 104.644 ms, Queueing time: mean = 104.232 us, max = 228.583 us, min = 11.443 us, total = 250.990 ms InternalKVGcsService.grpc_server.InternalKVPut - 2408 total (0 active), Execution time: mean = 569.748 us, total = 1.372 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2399 total (0 active), Execution time: mean = 35.969 us, total = 86.289 ms, Queueing time: mean = 118.955 us, max = 1.498 ms, min = 20.329 us, total = 285.374 ms NodeInfoGcsService.grpc_server.CheckAlive - 2399 total (0 active), Execution time: mean = 598.769 us, total = 1.436 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2001 total (1 active), Execution time: mean = 9.553 us, total = 19.115 ms, Queueing time: mean = 76.062 us, max = 391.368 us, min = 11.350 us, total = 152.201 ms HealthCheck - 1997 total (0 active), Execution time: mean = 7.742 us, total = 15.461 ms, Queueing time: mean = 103.272 us, max = 399.856 us, min = 16.806 us, total = 206.235 ms GcsInMemoryStore.Get - 1266 total (0 active), Execution time: mean = 21.102 us, total = 26.715 ms, Queueing time: mean = 13.543 us, max = 128.508 us, min = 3.701 us, total = 17.146 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1265 total (0 active), Execution time: mean = 57.614 us, total = 72.881 ms, Queueing time: mean = 105.745 us, max = 269.544 us, min = 13.461 us, total = 133.767 ms InternalKVGcsService.grpc_server.InternalKVGet - 1265 total (0 active), Execution time: mean = 612.870 us, total = 775.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1197 total (0 active), Execution time: mean = 92.880 us, total = 111.178 ms, Queueing time: mean = 117.877 us, max = 296.469 us, min = 14.122 us, total = 141.099 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1197 total (0 active), Execution time: mean = 685.758 us, total = 820.853 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 600 total (1 active), Execution time: mean = 1.431 ms, total = 858.606 ms, Queueing time: mean = 72.827 us, max = 323.875 us, min = -0.000 s, total = 43.696 ms GCSServer.deadline_timer.debug_state_event_stats_print - 100 total (1 active, 1 running), Execution time: mean = 1.576 ms, total = 157.577 ms, Queueing time: mean = 64.074 us, max = 182.019 us, min = 12.907 us, total = 6.407 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:41:53,968 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 277021 total (1 active) Queueing time: mean = 49.887 us, max = 7.565 ms, min = 1.647 us, total = 13.820 s Execution time: mean = 256.433 us, total = 71.037 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 125914 total (0 active), Execution time: mean = 530.348 us, total = 66.778 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 125914 total (0 active), Execution time: mean = 28.128 us, total = 3.542 s, Queueing time: mean = 106.962 us, max = 7.565 ms, min = 3.372 us, total = 13.468 s event_loop_lag_probe - 23991 total (0 active), Execution time: mean = 29.322 us, total = 703.472 ms, Queueing time: mean = 10.440 us, max = 4.983 ms, min = 1.647 us, total = 250.470 ms GcsTaskManager.GcJobSummary - 1201 total (1 active), Execution time: mean = 11.641 us, total = 13.981 ms, Queueing time: mean = 84.210 us, max = 265.885 us, min = 16.725 us, total = 101.137 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:41:53,969 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29523 total (25 active) Queueing time: mean = 19.670 us, max = 333.684 us, min = 2.330 us, total = 580.723 ms Execution time: mean = 4.824 s, total = 142420.289 s Event stats: event_loop_lag_probe - 23991 total (0 active), Execution time: mean = 31.821 us, total = 763.410 ms, Queueing time: mean = 11.725 us, max = 291.078 us, min = 2.330 us, total = 281.298 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1492 total (0 active), Execution time: mean = 43.641 us, total = 65.113 ms, Queueing time: mean = 104.945 us, max = 333.684 us, min = 6.593 us, total = 156.578 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1492 total (24 active), Execution time: mean = 95.455 s, total = 142418.476 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1238 total (0 active), Execution time: mean = 117.012 us, total = 144.861 ms, Queueing time: mean = 111.205 us, max = 249.252 us, min = 16.251 us, total = 137.671 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1238 total (0 active), Execution time: mean = 653.570 us, total = 809.120 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 670.588 us, total = 14.082 ms, Queueing time: mean = 83.734 us, max = 153.556 us, min = 48.361 us, total = 1.758 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:41:53,969 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24019 total (0 active) Queueing time: mean = 9.961 us, max = 1.423 ms, min = 568.000 ns, total = 239.264 ms Execution time: mean = 27.039 us, total = 649.450 ms Event stats: event_loop_lag_probe - 23993 total (0 active), Execution time: mean = 27.004 us, total = 647.901 ms, Queueing time: mean = 9.894 us, max = 1.423 ms, min = 1.850 us, total = 237.385 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:42:53,969 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1209 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1251 - cumulative published bytes: 994980 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:42:53,970 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 73003 total (4 active) Queueing time: mean = 76.186 us, max = 279.303 ms, min = -0.000 s, total = 5.562 s Execution time: mean = 246.070 us, total = 17.964 s Event stats: event_loop_lag_probe - 24232 total (0 active), Execution time: mean = 24.895 us, total = 603.259 ms, Queueing time: mean = 10.028 us, max = 13.097 ms, min = 1.994 us, total = 243.008 ms GcsInMemoryStore.Put - 9736 total (0 active), Execution time: mean = 34.221 us, total = 333.177 ms, Queueing time: mean = 246.871 us, max = 278.545 ms, min = 2.025 us, total = 2.404 s RayletLoadPulled - 6059 total (1 active), Execution time: mean = 249.465 us, total = 1.512 s, Queueing time: mean = 75.382 us, max = 472.090 us, min = -0.000 s, total = 456.742 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6056 total (0 active), Execution time: mean = 77.513 us, total = 469.419 ms, Queueing time: mean = 104.077 us, max = 2.744 ms, min = 11.910 us, total = 630.291 ms NodeManagerService.grpc_client.GetResourceLoad - 6056 total (0 active), Execution time: mean = 1.498 ms, total = 9.073 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2432 total (0 active), Execution time: mean = 43.382 us, total = 105.505 ms, Queueing time: mean = 104.163 us, max = 228.583 us, min = 11.443 us, total = 253.325 ms InternalKVGcsService.grpc_server.InternalKVPut - 2432 total (0 active), Execution time: mean = 569.662 us, total = 1.385 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2423 total (0 active), Execution time: mean = 35.948 us, total = 87.101 ms, Queueing time: mean = 118.928 us, max = 1.498 ms, min = 20.329 us, total = 288.163 ms NodeInfoGcsService.grpc_server.CheckAlive - 2423 total (0 active), Execution time: mean = 598.660 us, total = 1.451 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2021 total (1 active), Execution time: mean = 9.578 us, total = 19.357 ms, Queueing time: mean = 76.069 us, max = 391.368 us, min = 11.350 us, total = 153.736 ms HealthCheck - 2017 total (0 active), Execution time: mean = 7.742 us, total = 15.616 ms, Queueing time: mean = 103.335 us, max = 399.856 us, min = 16.806 us, total = 208.427 ms GcsInMemoryStore.Get - 1278 total (0 active), Execution time: mean = 21.083 us, total = 26.944 ms, Queueing time: mean = 13.535 us, max = 128.508 us, min = 3.701 us, total = 17.298 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1277 total (0 active), Execution time: mean = 57.722 us, total = 73.711 ms, Queueing time: mean = 105.705 us, max = 269.544 us, min = 13.461 us, total = 134.986 ms InternalKVGcsService.grpc_server.InternalKVGet - 1277 total (0 active), Execution time: mean = 612.792 us, total = 782.536 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1209 total (0 active), Execution time: mean = 92.835 us, total = 112.237 ms, Queueing time: mean = 117.838 us, max = 296.469 us, min = 14.122 us, total = 142.466 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1209 total (0 active), Execution time: mean = 685.739 us, total = 829.058 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 606 total (1 active), Execution time: mean = 1.429 ms, total = 865.820 ms, Queueing time: mean = 72.681 us, max = 323.875 us, min = -0.000 s, total = 44.045 ms GCSServer.deadline_timer.debug_state_event_stats_print - 101 total (1 active, 1 running), Execution time: mean = 1.576 ms, total = 159.166 ms, Queueing time: mean = 63.639 us, max = 182.019 us, min = 12.907 us, total = 6.428 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:42:53,971 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 279793 total (1 active) Queueing time: mean = 49.909 us, max = 7.565 ms, min = 1.647 us, total = 13.964 s Execution time: mean = 256.490 us, total = 71.764 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 127174 total (0 active), Execution time: mean = 530.460 us, total = 67.461 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 127174 total (0 active), Execution time: mean = 28.138 us, total = 3.578 s, Queueing time: mean = 107.010 us, max = 7.565 ms, min = 3.372 us, total = 13.609 s event_loop_lag_probe - 24231 total (0 active), Execution time: mean = 29.330 us, total = 710.689 ms, Queueing time: mean = 10.438 us, max = 4.983 ms, min = 1.647 us, total = 252.932 ms GcsTaskManager.GcJobSummary - 1213 total (1 active), Execution time: mean = 11.650 us, total = 14.132 ms, Queueing time: mean = 84.273 us, max = 265.885 us, min = 16.725 us, total = 102.223 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:42:53,971 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 29815 total (25 active) Queueing time: mean = 19.670 us, max = 333.684 us, min = 2.330 us, total = 586.457 ms Execution time: mean = 4.779 s, total = 142480.494 s Event stats: event_loop_lag_probe - 24231 total (0 active), Execution time: mean = 31.832 us, total = 771.323 ms, Queueing time: mean = 11.727 us, max = 291.078 us, min = 2.330 us, total = 284.165 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1505 total (0 active), Execution time: mean = 43.755 us, total = 65.851 ms, Queueing time: mean = 104.840 us, max = 333.684 us, min = 6.593 us, total = 157.784 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1505 total (24 active), Execution time: mean = 94.670 s, total = 142478.662 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1251 total (0 active), Execution time: mean = 116.916 us, total = 146.262 ms, Queueing time: mean = 111.376 us, max = 293.835 us, min = 16.251 us, total = 139.332 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1251 total (0 active), Execution time: mean = 653.314 us, total = 817.296 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 670.588 us, total = 14.082 ms, Queueing time: mean = 83.734 us, max = 153.556 us, min = 48.361 us, total = 1.758 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:42:53,971 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24259 total (0 active) Queueing time: mean = 9.959 us, max = 1.423 ms, min = 568.000 ns, total = 241.603 ms Execution time: mean = 27.037 us, total = 655.894 ms Event stats: event_loop_lag_probe - 24233 total (0 active), Execution time: mean = 27.002 us, total = 654.345 ms, Queueing time: mean = 9.892 us, max = 1.423 ms, min = 1.850 us, total = 239.724 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:43:53,971 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1221 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1264 - cumulative published bytes: 1004842 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:43:53,972 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 73722 total (4 active) Queueing time: mean = 75.894 us, max = 279.303 ms, min = -0.000 s, total = 5.595 s Execution time: mean = 246.115 us, total = 18.144 s Event stats: event_loop_lag_probe - 24472 total (0 active), Execution time: mean = 24.944 us, total = 610.431 ms, Queueing time: mean = 10.039 us, max = 13.097 ms, min = 1.994 us, total = 245.685 ms GcsInMemoryStore.Put - 9832 total (0 active), Execution time: mean = 33.939 us, total = 333.685 ms, Queueing time: mean = 244.912 us, max = 278.545 ms, min = 2.025 us, total = 2.408 s RayletLoadPulled - 6119 total (1 active), Execution time: mean = 249.599 us, total = 1.527 s, Queueing time: mean = 75.470 us, max = 472.090 us, min = -0.000 s, total = 461.801 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6116 total (0 active), Execution time: mean = 77.560 us, total = 474.355 ms, Queueing time: mean = 104.203 us, max = 2.744 ms, min = 11.910 us, total = 637.308 ms NodeManagerService.grpc_client.GetResourceLoad - 6116 total (0 active), Execution time: mean = 1.499 ms, total = 9.166 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2456 total (0 active), Execution time: mean = 43.383 us, total = 106.550 ms, Queueing time: mean = 104.308 us, max = 269.678 us, min = 11.443 us, total = 256.180 ms InternalKVGcsService.grpc_server.InternalKVPut - 2456 total (0 active), Execution time: mean = 569.550 us, total = 1.399 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2447 total (0 active), Execution time: mean = 35.955 us, total = 87.981 ms, Queueing time: mean = 119.135 us, max = 1.498 ms, min = 20.329 us, total = 291.524 ms NodeInfoGcsService.grpc_server.CheckAlive - 2447 total (0 active), Execution time: mean = 598.643 us, total = 1.465 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2041 total (1 active), Execution time: mean = 9.599 us, total = 19.592 ms, Queueing time: mean = 76.150 us, max = 391.368 us, min = 11.350 us, total = 155.422 ms HealthCheck - 2037 total (0 active), Execution time: mean = 7.752 us, total = 15.790 ms, Queueing time: mean = 103.467 us, max = 399.856 us, min = 16.806 us, total = 210.763 ms GcsInMemoryStore.Get - 1290 total (0 active), Execution time: mean = 21.050 us, total = 27.154 ms, Queueing time: mean = 13.525 us, max = 128.508 us, min = 3.701 us, total = 17.448 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1289 total (0 active), Execution time: mean = 57.763 us, total = 74.457 ms, Queueing time: mean = 105.824 us, max = 269.544 us, min = 13.461 us, total = 136.407 ms InternalKVGcsService.grpc_server.InternalKVGet - 1289 total (0 active), Execution time: mean = 612.548 us, total = 789.574 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1221 total (0 active), Execution time: mean = 92.945 us, total = 113.485 ms, Queueing time: mean = 117.877 us, max = 296.469 us, min = 14.122 us, total = 143.928 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1221 total (0 active), Execution time: mean = 685.982 us, total = 837.584 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 612 total (1 active), Execution time: mean = 1.429 ms, total = 874.699 ms, Queueing time: mean = 73.035 us, max = 323.875 us, min = -0.000 s, total = 44.698 ms GCSServer.deadline_timer.debug_state_event_stats_print - 102 total (1 active, 1 running), Execution time: mean = 1.581 ms, total = 161.286 ms, Queueing time: mean = 64.319 us, max = 182.019 us, min = 12.907 us, total = 6.561 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:43:53,973 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 282565 total (1 active) Queueing time: mean = 49.916 us, max = 7.565 ms, min = 1.647 us, total = 14.105 s Execution time: mean = 256.538 us, total = 72.489 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 128434 total (0 active), Execution time: mean = 530.543 us, total = 68.140 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 128434 total (0 active), Execution time: mean = 28.160 us, total = 3.617 s, Queueing time: mean = 107.025 us, max = 7.565 ms, min = 3.372 us, total = 13.746 s event_loop_lag_probe - 24471 total (0 active), Execution time: mean = 29.331 us, total = 717.762 ms, Queueing time: mean = 10.443 us, max = 4.983 ms, min = 1.647 us, total = 255.552 ms GcsTaskManager.GcJobSummary - 1225 total (1 active), Execution time: mean = 11.664 us, total = 14.289 ms, Queueing time: mean = 84.220 us, max = 265.885 us, min = 16.725 us, total = 103.170 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:43:53,973 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30107 total (25 active) Queueing time: mean = 19.675 us, max = 333.684 us, min = 2.330 us, total = 592.356 ms Execution time: mean = 4.734 s, total = 142540.690 s Event stats: event_loop_lag_probe - 24471 total (0 active), Execution time: mean = 31.856 us, total = 779.543 ms, Queueing time: mean = 11.742 us, max = 291.078 us, min = 2.330 us, total = 287.347 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1518 total (0 active), Execution time: mean = 43.797 us, total = 66.485 ms, Queueing time: mean = 104.817 us, max = 333.684 us, min = 6.593 us, total = 159.112 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1518 total (24 active), Execution time: mean = 93.899 s, total = 142538.839 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1264 total (0 active), Execution time: mean = 117.016 us, total = 147.908 ms, Queueing time: mean = 111.330 us, max = 293.835 us, min = 16.251 us, total = 140.721 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1264 total (0 active), Execution time: mean = 653.224 us, total = 825.675 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 670.588 us, total = 14.082 ms, Queueing time: mean = 83.734 us, max = 153.556 us, min = 48.361 us, total = 1.758 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:43:53,973 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24499 total (0 active) Queueing time: mean = 9.976 us, max = 1.423 ms, min = 568.000 ns, total = 244.398 ms Execution time: mean = 27.081 us, total = 663.464 ms Event stats: event_loop_lag_probe - 24473 total (0 active), Execution time: mean = 27.047 us, total = 661.915 ms, Queueing time: mean = 9.910 us, max = 1.423 ms, min = 1.850 us, total = 242.519 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:44:53,973 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1233 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1276 - cumulative published bytes: 1014670 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:44:53,974 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 74441 total (4 active) Queueing time: mean = 75.553 us, max = 279.303 ms, min = -0.000 s, total = 5.624 s Execution time: mean = 245.959 us, total = 18.309 s Event stats: event_loop_lag_probe - 24712 total (0 active), Execution time: mean = 24.981 us, total = 617.337 ms, Queueing time: mean = 10.047 us, max = 13.097 ms, min = 1.994 us, total = 248.290 ms GcsInMemoryStore.Put - 9928 total (0 active), Execution time: mean = 33.657 us, total = 334.151 ms, Queueing time: mean = 242.976 us, max = 278.545 ms, min = 2.025 us, total = 2.412 s RayletLoadPulled - 6179 total (1 active), Execution time: mean = 249.461 us, total = 1.541 s, Queueing time: mean = 75.443 us, max = 472.090 us, min = -0.000 s, total = 466.162 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6176 total (0 active), Execution time: mean = 77.529 us, total = 478.820 ms, Queueing time: mean = 104.110 us, max = 2.744 ms, min = 11.910 us, total = 642.980 ms NodeManagerService.grpc_client.GetResourceLoad - 6176 total (0 active), Execution time: mean = 1.498 ms, total = 9.250 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2480 total (0 active), Execution time: mean = 43.340 us, total = 107.482 ms, Queueing time: mean = 104.243 us, max = 269.678 us, min = 11.443 us, total = 258.523 ms InternalKVGcsService.grpc_server.InternalKVPut - 2480 total (0 active), Execution time: mean = 569.196 us, total = 1.412 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2471 total (0 active), Execution time: mean = 35.933 us, total = 88.791 ms, Queueing time: mean = 118.922 us, max = 1.498 ms, min = 20.329 us, total = 293.857 ms NodeInfoGcsService.grpc_server.CheckAlive - 2471 total (0 active), Execution time: mean = 598.283 us, total = 1.478 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2061 total (1 active), Execution time: mean = 9.606 us, total = 19.799 ms, Queueing time: mean = 76.196 us, max = 391.368 us, min = 11.350 us, total = 157.039 ms HealthCheck - 2057 total (0 active), Execution time: mean = 7.752 us, total = 15.947 ms, Queueing time: mean = 103.675 us, max = 564.605 us, min = 16.806 us, total = 213.259 ms GcsInMemoryStore.Get - 1302 total (0 active), Execution time: mean = 21.026 us, total = 27.376 ms, Queueing time: mean = 13.507 us, max = 128.508 us, min = 3.701 us, total = 17.586 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1301 total (0 active), Execution time: mean = 57.796 us, total = 75.193 ms, Queueing time: mean = 105.950 us, max = 269.544 us, min = 13.461 us, total = 137.841 ms InternalKVGcsService.grpc_server.InternalKVGet - 1301 total (0 active), Execution time: mean = 612.455 us, total = 796.804 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1233 total (0 active), Execution time: mean = 92.872 us, total = 114.511 ms, Queueing time: mean = 117.804 us, max = 296.469 us, min = 14.122 us, total = 145.253 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1233 total (0 active), Execution time: mean = 685.758 us, total = 845.540 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 618 total (1 active), Execution time: mean = 1.429 ms, total = 882.990 ms, Queueing time: mean = 73.064 us, max = 323.875 us, min = -0.000 s, total = 45.153 ms GCSServer.deadline_timer.debug_state_event_stats_print - 103 total (1 active, 1 running), Execution time: mean = 1.583 ms, total = 163.006 ms, Queueing time: mean = 64.741 us, max = 182.019 us, min = 12.907 us, total = 6.668 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:44:53,974 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 285337 total (1 active) Queueing time: mean = 49.915 us, max = 7.565 ms, min = 1.647 us, total = 14.243 s Execution time: mean = 256.569 us, total = 73.209 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 129694 total (0 active), Execution time: mean = 530.589 us, total = 68.814 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 129694 total (0 active), Execution time: mean = 28.187 us, total = 3.656 s, Queueing time: mean = 107.026 us, max = 7.565 ms, min = 3.372 us, total = 13.881 s event_loop_lag_probe - 24711 total (0 active), Execution time: mean = 29.318 us, total = 724.488 ms, Queueing time: mean = 10.438 us, max = 4.983 ms, min = 1.647 us, total = 257.931 ms GcsTaskManager.GcJobSummary - 1237 total (1 active), Execution time: mean = 11.646 us, total = 14.406 ms, Queueing time: mean = 84.145 us, max = 265.885 us, min = 16.725 us, total = 104.088 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:44:53,974 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30395 total (25 active) Queueing time: mean = 19.683 us, max = 333.684 us, min = 2.330 us, total = 598.254 ms Execution time: mean = 4.692 s, total = 142600.862 s Event stats: event_loop_lag_probe - 24711 total (0 active), Execution time: mean = 31.884 us, total = 787.896 ms, Queueing time: mean = 11.758 us, max = 291.078 us, min = 2.330 us, total = 290.541 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1530 total (0 active), Execution time: mean = 43.873 us, total = 67.126 ms, Queueing time: mean = 104.827 us, max = 333.684 us, min = 6.593 us, total = 160.385 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1530 total (24 active), Execution time: mean = 93.202 s, total = 142598.993 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1276 total (0 active), Execution time: mean = 116.970 us, total = 149.253 ms, Queueing time: mean = 111.405 us, max = 293.835 us, min = 16.251 us, total = 142.153 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1276 total (0 active), Execution time: mean = 653.481 us, total = 833.842 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 670.588 us, total = 14.082 ms, Queueing time: mean = 83.734 us, max = 153.556 us, min = 48.361 us, total = 1.758 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:44:53,974 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24739 total (0 active) Queueing time: mean = 9.986 us, max = 1.423 ms, min = 568.000 ns, total = 247.046 ms Execution time: mean = 27.106 us, total = 670.579 ms Event stats: event_loop_lag_probe - 24713 total (0 active), Execution time: mean = 27.072 us, total = 669.030 ms, Queueing time: mean = 9.921 us, max = 1.423 ms, min = 1.850 us, total = 245.166 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:45:53,975 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1245 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1289 - cumulative published bytes: 1024532 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:45:53,977 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 75160 total (4 active) Queueing time: mean = 75.230 us, max = 279.303 ms, min = -0.000 s, total = 5.654 s Execution time: mean = 245.942 us, total = 18.485 s Event stats: event_loop_lag_probe - 24952 total (0 active), Execution time: mean = 24.982 us, total = 623.345 ms, Queueing time: mean = 10.038 us, max = 13.097 ms, min = 1.994 us, total = 250.477 ms GcsInMemoryStore.Put - 10024 total (0 active), Execution time: mean = 33.395 us, total = 334.752 ms, Queueing time: mean = 241.103 us, max = 278.545 ms, min = 2.025 us, total = 2.417 s RayletLoadPulled - 6239 total (1 active), Execution time: mean = 249.375 us, total = 1.556 s, Queueing time: mean = 75.469 us, max = 472.090 us, min = -0.000 s, total = 470.853 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6236 total (0 active), Execution time: mean = 77.514 us, total = 483.377 ms, Queueing time: mean = 104.104 us, max = 2.744 ms, min = 11.910 us, total = 649.190 ms NodeManagerService.grpc_client.GetResourceLoad - 6236 total (0 active), Execution time: mean = 1.498 ms, total = 9.341 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2504 total (0 active), Execution time: mean = 43.314 us, total = 108.458 ms, Queueing time: mean = 104.245 us, max = 269.678 us, min = 11.443 us, total = 261.030 ms InternalKVGcsService.grpc_server.InternalKVPut - 2504 total (0 active), Execution time: mean = 569.404 us, total = 1.426 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2495 total (0 active), Execution time: mean = 35.993 us, total = 89.803 ms, Queueing time: mean = 118.862 us, max = 1.498 ms, min = 20.329 us, total = 296.559 ms NodeInfoGcsService.grpc_server.CheckAlive - 2495 total (0 active), Execution time: mean = 598.269 us, total = 1.493 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2081 total (1 active), Execution time: mean = 9.592 us, total = 19.962 ms, Queueing time: mean = 76.199 us, max = 391.368 us, min = 11.350 us, total = 158.571 ms HealthCheck - 2077 total (0 active), Execution time: mean = 7.751 us, total = 16.098 ms, Queueing time: mean = 103.703 us, max = 564.605 us, min = 16.806 us, total = 215.392 ms GcsInMemoryStore.Get - 1314 total (0 active), Execution time: mean = 21.025 us, total = 27.627 ms, Queueing time: mean = 13.512 us, max = 128.508 us, min = 3.701 us, total = 17.755 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1313 total (0 active), Execution time: mean = 57.891 us, total = 76.010 ms, Queueing time: mean = 105.908 us, max = 269.544 us, min = 13.461 us, total = 139.057 ms InternalKVGcsService.grpc_server.InternalKVGet - 1313 total (0 active), Execution time: mean = 612.436 us, total = 804.128 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1245 total (0 active), Execution time: mean = 92.960 us, total = 115.735 ms, Queueing time: mean = 117.944 us, max = 296.469 us, min = 14.122 us, total = 146.840 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1245 total (0 active), Execution time: mean = 686.053 us, total = 854.136 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 624 total (1 active), Execution time: mean = 1.429 ms, total = 891.389 ms, Queueing time: mean = 73.159 us, max = 323.875 us, min = -0.000 s, total = 45.651 ms GCSServer.deadline_timer.debug_state_event_stats_print - 104 total (1 active, 1 running), Execution time: mean = 1.582 ms, total = 164.553 ms, Queueing time: mean = 64.754 us, max = 182.019 us, min = 12.907 us, total = 6.734 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:45:53,977 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 288107 total (1 active) Queueing time: mean = 49.929 us, max = 7.565 ms, min = 1.647 us, total = 14.385 s Execution time: mean = 256.643 us, total = 73.941 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 130953 total (0 active), Execution time: mean = 530.745 us, total = 69.503 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 130953 total (0 active), Execution time: mean = 28.195 us, total = 3.692 s, Queueing time: mean = 107.058 us, max = 7.565 ms, min = 3.372 us, total = 14.020 s event_loop_lag_probe - 24951 total (0 active), Execution time: mean = 29.307 us, total = 731.227 ms, Queueing time: mean = 10.432 us, max = 4.983 ms, min = 1.647 us, total = 260.277 ms GcsTaskManager.GcJobSummary - 1249 total (1 active), Execution time: mean = 11.647 us, total = 14.547 ms, Queueing time: mean = 84.134 us, max = 265.885 us, min = 16.725 us, total = 105.084 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:45:53,977 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30687 total (25 active) Queueing time: mean = 19.688 us, max = 333.684 us, min = 2.330 us, total = 604.166 ms Execution time: mean = 4.649 s, total = 142660.957 s Event stats: event_loop_lag_probe - 24951 total (0 active), Execution time: mean = 31.898 us, total = 795.887 ms, Queueing time: mean = 11.763 us, max = 291.078 us, min = 2.330 us, total = 293.503 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1543 total (0 active), Execution time: mean = 43.935 us, total = 67.792 ms, Queueing time: mean = 104.976 us, max = 333.684 us, min = 6.593 us, total = 161.977 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1543 total (24 active), Execution time: mean = 92.456 s, total = 142659.069 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1289 total (0 active), Execution time: mean = 117.108 us, total = 150.952 ms, Queueing time: mean = 111.334 us, max = 293.835 us, min = 16.251 us, total = 143.510 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1289 total (0 active), Execution time: mean = 653.635 us, total = 842.536 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 21 total (1 active), Execution time: mean = 670.588 us, total = 14.082 ms, Queueing time: mean = 83.734 us, max = 153.556 us, min = 48.361 us, total = 1.758 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:45:53,977 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 24978 total (0 active) Queueing time: mean = 9.981 us, max = 1.423 ms, min = 568.000 ns, total = 249.298 ms Execution time: mean = 27.102 us, total = 676.950 ms Event stats: event_loop_lag_probe - 24952 total (0 active), Execution time: mean = 27.068 us, total = 675.401 ms, Queueing time: mean = 9.916 us, max = 1.423 ms, min = 1.850 us, total = 247.419 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:46:53,978 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1257 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1302 - cumulative published bytes: 1034394 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:46:53,979 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 75879 total (4 active) Queueing time: mean = 74.918 us, max = 279.303 ms, min = -0.000 s, total = 5.685 s Execution time: mean = 245.966 us, total = 18.664 s Event stats: event_loop_lag_probe - 25192 total (0 active), Execution time: mean = 24.984 us, total = 629.404 ms, Queueing time: mean = 10.029 us, max = 13.097 ms, min = 1.994 us, total = 252.649 ms GcsInMemoryStore.Put - 10120 total (0 active), Execution time: mean = 33.128 us, total = 335.256 ms, Queueing time: mean = 239.262 us, max = 278.545 ms, min = 2.025 us, total = 2.421 s RayletLoadPulled - 6299 total (1 active), Execution time: mean = 249.417 us, total = 1.571 s, Queueing time: mean = 75.496 us, max = 472.090 us, min = -0.000 s, total = 475.549 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6296 total (0 active), Execution time: mean = 77.564 us, total = 488.343 ms, Queueing time: mean = 104.162 us, max = 2.744 ms, min = 11.910 us, total = 655.803 ms NodeManagerService.grpc_client.GetResourceLoad - 6296 total (0 active), Execution time: mean = 1.498 ms, total = 9.432 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2528 total (0 active), Execution time: mean = 43.314 us, total = 109.498 ms, Queueing time: mean = 104.341 us, max = 269.678 us, min = 11.443 us, total = 263.774 ms InternalKVGcsService.grpc_server.InternalKVPut - 2528 total (0 active), Execution time: mean = 569.720 us, total = 1.440 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2519 total (0 active), Execution time: mean = 36.060 us, total = 90.836 ms, Queueing time: mean = 118.770 us, max = 1.498 ms, min = 20.329 us, total = 299.181 ms NodeInfoGcsService.grpc_server.CheckAlive - 2519 total (0 active), Execution time: mean = 598.232 us, total = 1.507 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2101 total (1 active), Execution time: mean = 9.584 us, total = 20.136 ms, Queueing time: mean = 76.111 us, max = 391.368 us, min = 11.350 us, total = 159.910 ms HealthCheck - 2097 total (0 active), Execution time: mean = 7.747 us, total = 16.246 ms, Queueing time: mean = 103.753 us, max = 564.605 us, min = 16.806 us, total = 217.570 ms GcsInMemoryStore.Get - 1326 total (0 active), Execution time: mean = 21.024 us, total = 27.878 ms, Queueing time: mean = 13.515 us, max = 128.508 us, min = 3.701 us, total = 17.921 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1325 total (0 active), Execution time: mean = 57.960 us, total = 76.797 ms, Queueing time: mean = 105.946 us, max = 269.544 us, min = 13.461 us, total = 140.378 ms InternalKVGcsService.grpc_server.InternalKVGet - 1325 total (0 active), Execution time: mean = 612.448 us, total = 811.493 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1257 total (0 active), Execution time: mean = 93.150 us, total = 117.090 ms, Queueing time: mean = 118.042 us, max = 296.469 us, min = 14.122 us, total = 148.379 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1257 total (0 active), Execution time: mean = 686.559 us, total = 863.005 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 630 total (1 active), Execution time: mean = 1.428 ms, total = 899.698 ms, Queueing time: mean = 73.115 us, max = 323.875 us, min = -0.000 s, total = 46.063 ms GCSServer.deadline_timer.debug_state_event_stats_print - 105 total (1 active, 1 running), Execution time: mean = 1.594 ms, total = 167.420 ms, Queueing time: mean = 64.815 us, max = 182.019 us, min = 12.907 us, total = 6.806 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:46:53,979 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 290877 total (1 active) Queueing time: mean = 49.937 us, max = 7.565 ms, min = 1.647 us, total = 14.525 s Execution time: mean = 256.747 us, total = 74.682 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 132212 total (0 active), Execution time: mean = 530.947 us, total = 70.198 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 132212 total (0 active), Execution time: mean = 28.220 us, total = 3.731 s, Queueing time: mean = 107.073 us, max = 7.565 ms, min = 3.372 us, total = 14.156 s event_loop_lag_probe - 25191 total (0 active), Execution time: mean = 29.320 us, total = 738.589 ms, Queueing time: mean = 10.437 us, max = 4.983 ms, min = 1.647 us, total = 262.916 ms GcsTaskManager.GcJobSummary - 1261 total (1 active), Execution time: mean = 11.662 us, total = 14.706 ms, Queueing time: mean = 84.100 us, max = 265.885 us, min = 16.725 us, total = 106.051 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:46:53,979 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 30990 total (25 active) Queueing time: mean = 19.703 us, max = 333.684 us, min = 2.330 us, total = 610.606 ms Execution time: mean = 4.702 s, total = 145721.172 s Event stats: event_loop_lag_probe - 25191 total (0 active), Execution time: mean = 31.907 us, total = 803.778 ms, Queueing time: mean = 11.767 us, max = 291.078 us, min = 2.330 us, total = 296.434 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1561 total (0 active), Execution time: mean = 43.901 us, total = 68.529 ms, Queueing time: mean = 104.892 us, max = 333.684 us, min = 6.593 us, total = 163.737 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1561 total (24 active), Execution time: mean = 93.350 s, total = 145719.264 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1302 total (0 active), Execution time: mean = 117.254 us, total = 152.665 ms, Queueing time: mean = 111.488 us, max = 293.835 us, min = 16.251 us, total = 145.157 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1302 total (0 active), Execution time: mean = 653.985 us, total = 851.489 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 647.697 us, total = 14.249 ms, Queueing time: mean = 84.597 us, max = 153.556 us, min = 48.361 us, total = 1.861 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:46:53,979 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25218 total (0 active) Queueing time: mean = 9.971 us, max = 1.423 ms, min = 568.000 ns, total = 251.459 ms Execution time: mean = 27.081 us, total = 682.936 ms Event stats: event_loop_lag_probe - 25192 total (0 active), Execution time: mean = 27.048 us, total = 681.387 ms, Queueing time: mean = 9.907 us, max = 1.423 ms, min = 1.850 us, total = 249.580 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:47:53,980 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1269 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1314 - cumulative published bytes: 1044222 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:47:53,981 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 76598 total (4 active) Queueing time: mean = 74.591 us, max = 279.303 ms, min = -0.000 s, total = 5.714 s Execution time: mean = 245.866 us, total = 18.833 s Event stats: event_loop_lag_probe - 25432 total (0 active), Execution time: mean = 25.001 us, total = 635.815 ms, Queueing time: mean = 10.028 us, max = 13.097 ms, min = 1.994 us, total = 255.023 ms GcsInMemoryStore.Put - 10216 total (0 active), Execution time: mean = 32.864 us, total = 335.740 ms, Queueing time: mean = 237.416 us, max = 278.545 ms, min = 2.025 us, total = 2.425 s RayletLoadPulled - 6359 total (1 active), Execution time: mean = 249.392 us, total = 1.586 s, Queueing time: mean = 75.531 us, max = 472.090 us, min = -0.000 s, total = 480.304 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6356 total (0 active), Execution time: mean = 77.550 us, total = 492.906 ms, Queueing time: mean = 104.095 us, max = 2.744 ms, min = 11.910 us, total = 661.626 ms NodeManagerService.grpc_client.GetResourceLoad - 6356 total (0 active), Execution time: mean = 1.497 ms, total = 9.517 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2552 total (0 active), Execution time: mean = 43.296 us, total = 110.491 ms, Queueing time: mean = 104.340 us, max = 269.678 us, min = 11.443 us, total = 266.277 ms InternalKVGcsService.grpc_server.InternalKVPut - 2552 total (0 active), Execution time: mean = 569.686 us, total = 1.454 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2543 total (0 active), Execution time: mean = 36.008 us, total = 91.567 ms, Queueing time: mean = 118.629 us, max = 1.498 ms, min = 20.329 us, total = 301.674 ms NodeInfoGcsService.grpc_server.CheckAlive - 2543 total (0 active), Execution time: mean = 597.839 us, total = 1.520 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2121 total (1 active), Execution time: mean = 9.573 us, total = 20.305 ms, Queueing time: mean = 76.070 us, max = 391.368 us, min = 11.350 us, total = 161.345 ms HealthCheck - 2117 total (0 active), Execution time: mean = 7.746 us, total = 16.399 ms, Queueing time: mean = 103.731 us, max = 564.605 us, min = 16.806 us, total = 219.598 ms GcsInMemoryStore.Get - 1338 total (0 active), Execution time: mean = 21.016 us, total = 28.120 ms, Queueing time: mean = 13.518 us, max = 128.508 us, min = 3.701 us, total = 18.087 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1337 total (0 active), Execution time: mean = 58.015 us, total = 77.567 ms, Queueing time: mean = 105.929 us, max = 269.544 us, min = 13.461 us, total = 141.627 ms InternalKVGcsService.grpc_server.InternalKVGet - 1337 total (0 active), Execution time: mean = 612.443 us, total = 818.836 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1269 total (0 active), Execution time: mean = 93.315 us, total = 118.416 ms, Queueing time: mean = 118.020 us, max = 296.469 us, min = 14.122 us, total = 149.767 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1269 total (0 active), Execution time: mean = 686.739 us, total = 871.472 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 636 total (1 active), Execution time: mean = 1.429 ms, total = 908.836 ms, Queueing time: mean = 73.121 us, max = 323.875 us, min = -0.000 s, total = 46.505 ms GCSServer.deadline_timer.debug_state_event_stats_print - 106 total (1 active, 1 running), Execution time: mean = 1.597 ms, total = 169.301 ms, Queueing time: mean = 64.771 us, max = 182.019 us, min = 12.907 us, total = 6.866 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:47:53,981 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 293646 total (1 active) Queueing time: mean = 49.927 us, max = 7.565 ms, min = 1.647 us, total = 14.661 s Execution time: mean = 256.720 us, total = 75.385 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 133471 total (0 active), Execution time: mean = 530.874 us, total = 70.856 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 133471 total (0 active), Execution time: mean = 28.230 us, total = 3.768 s, Queueing time: mean = 107.051 us, max = 7.565 ms, min = 3.372 us, total = 14.288 s event_loop_lag_probe - 25430 total (0 active), Execution time: mean = 29.332 us, total = 745.920 ms, Queueing time: mean = 10.438 us, max = 4.983 ms, min = 1.647 us, total = 265.438 ms GcsTaskManager.GcJobSummary - 1273 total (1 active), Execution time: mean = 11.663 us, total = 14.847 ms, Queueing time: mean = 84.030 us, max = 265.885 us, min = 16.725 us, total = 106.970 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:47:53,981 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31277 total (25 active) Queueing time: mean = 19.691 us, max = 333.684 us, min = 2.330 us, total = 615.874 ms Execution time: mean = 4.661 s, total = 145781.353 s Event stats: event_loop_lag_probe - 25430 total (0 active), Execution time: mean = 31.916 us, total = 811.635 ms, Queueing time: mean = 11.769 us, max = 291.078 us, min = 2.330 us, total = 299.289 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1573 total (0 active), Execution time: mean = 43.947 us, total = 69.128 ms, Queueing time: mean = 104.816 us, max = 333.684 us, min = 6.593 us, total = 164.875 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1573 total (24 active), Execution time: mean = 92.676 s, total = 145779.428 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1314 total (0 active), Execution time: mean = 117.224 us, total = 154.032 ms, Queueing time: mean = 111.439 us, max = 293.835 us, min = 16.251 us, total = 146.431 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1314 total (0 active), Execution time: mean = 653.861 us, total = 859.174 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 647.697 us, total = 14.249 ms, Queueing time: mean = 84.597 us, max = 153.556 us, min = 48.361 us, total = 1.861 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:47:53,981 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25458 total (0 active) Queueing time: mean = 9.969 us, max = 1.423 ms, min = 568.000 ns, total = 253.788 ms Execution time: mean = 27.074 us, total = 689.256 ms Event stats: event_loop_lag_probe - 25432 total (0 active), Execution time: mean = 27.041 us, total = 687.707 ms, Queueing time: mean = 9.905 us, max = 1.423 ms, min = 1.850 us, total = 251.909 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:48:53,981 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1281 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1327 - cumulative published bytes: 1054084 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:48:53,982 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 77317 total (4 active) Queueing time: mean = 74.271 us, max = 279.303 ms, min = -0.000 s, total = 5.742 s Execution time: mean = 245.807 us, total = 19.005 s Event stats: event_loop_lag_probe - 25672 total (0 active), Execution time: mean = 25.004 us, total = 641.900 ms, Queueing time: mean = 10.021 us, max = 13.097 ms, min = 1.994 us, total = 257.255 ms GcsInMemoryStore.Put - 10312 total (0 active), Execution time: mean = 32.606 us, total = 336.233 ms, Queueing time: mean = 235.635 us, max = 278.545 ms, min = 2.025 us, total = 2.430 s RayletLoadPulled - 6419 total (1 active), Execution time: mean = 249.343 us, total = 1.601 s, Queueing time: mean = 75.568 us, max = 472.090 us, min = -0.000 s, total = 485.068 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6416 total (0 active), Execution time: mean = 77.552 us, total = 497.571 ms, Queueing time: mean = 103.958 us, max = 2.744 ms, min = 11.910 us, total = 666.995 ms NodeManagerService.grpc_client.GetResourceLoad - 6416 total (0 active), Execution time: mean = 1.497 ms, total = 9.605 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2576 total (0 active), Execution time: mean = 43.343 us, total = 111.652 ms, Queueing time: mean = 104.529 us, max = 479.211 us, min = 11.443 us, total = 269.266 ms InternalKVGcsService.grpc_server.InternalKVPut - 2576 total (0 active), Execution time: mean = 570.030 us, total = 1.468 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2567 total (0 active), Execution time: mean = 35.968 us, total = 92.331 ms, Queueing time: mean = 118.574 us, max = 1.498 ms, min = 20.329 us, total = 304.379 ms NodeInfoGcsService.grpc_server.CheckAlive - 2567 total (0 active), Execution time: mean = 597.368 us, total = 1.533 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2141 total (1 active), Execution time: mean = 9.561 us, total = 20.471 ms, Queueing time: mean = 75.935 us, max = 391.368 us, min = 11.350 us, total = 162.578 ms HealthCheck - 2137 total (0 active), Execution time: mean = 7.743 us, total = 16.548 ms, Queueing time: mean = 103.710 us, max = 564.605 us, min = 16.806 us, total = 221.628 ms GcsInMemoryStore.Get - 1350 total (0 active), Execution time: mean = 21.002 us, total = 28.353 ms, Queueing time: mean = 13.505 us, max = 128.508 us, min = 3.701 us, total = 18.231 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1349 total (0 active), Execution time: mean = 58.099 us, total = 78.375 ms, Queueing time: mean = 105.911 us, max = 269.544 us, min = 13.461 us, total = 142.874 ms InternalKVGcsService.grpc_server.InternalKVGet - 1349 total (0 active), Execution time: mean = 614.165 us, total = 828.509 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1281 total (0 active), Execution time: mean = 93.393 us, total = 119.636 ms, Queueing time: mean = 117.963 us, max = 296.469 us, min = 14.122 us, total = 151.110 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1281 total (0 active), Execution time: mean = 686.731 us, total = 879.702 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 642 total (1 active), Execution time: mean = 1.426 ms, total = 915.513 ms, Queueing time: mean = 72.846 us, max = 323.875 us, min = -0.000 s, total = 46.767 ms GCSServer.deadline_timer.debug_state_event_stats_print - 107 total (1 active, 1 running), Execution time: mean = 1.597 ms, total = 170.870 ms, Queueing time: mean = 65.189 us, max = 182.019 us, min = 12.907 us, total = 6.975 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:48:53,983 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 296416 total (1 active) Queueing time: mean = 49.920 us, max = 7.565 ms, min = 1.647 us, total = 14.797 s Execution time: mean = 256.724 us, total = 76.097 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 134730 total (0 active), Execution time: mean = 530.878 us, total = 71.525 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 134730 total (0 active), Execution time: mean = 28.235 us, total = 3.804 s, Queueing time: mean = 107.037 us, max = 7.565 ms, min = 3.372 us, total = 14.421 s event_loop_lag_probe - 25670 total (0 active), Execution time: mean = 29.330 us, total = 752.891 ms, Queueing time: mean = 10.434 us, max = 4.983 ms, min = 1.647 us, total = 267.845 ms GcsTaskManager.GcJobSummary - 1285 total (1 active), Execution time: mean = 11.672 us, total = 14.999 ms, Queueing time: mean = 84.087 us, max = 265.885 us, min = 16.725 us, total = 108.052 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:48:53,983 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31569 total (25 active) Queueing time: mean = 19.681 us, max = 333.684 us, min = 2.330 us, total = 621.317 ms Execution time: mean = 4.620 s, total = 145841.512 s Event stats: event_loop_lag_probe - 25670 total (0 active), Execution time: mean = 31.899 us, total = 818.845 ms, Queueing time: mean = 11.764 us, max = 291.078 us, min = 2.330 us, total = 301.974 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1586 total (0 active), Execution time: mean = 43.928 us, total = 69.669 ms, Queueing time: mean = 104.831 us, max = 333.684 us, min = 6.593 us, total = 166.262 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1586 total (24 active), Execution time: mean = 91.954 s, total = 145839.569 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1327 total (0 active), Execution time: mean = 117.090 us, total = 155.378 ms, Queueing time: mean = 111.381 us, max = 293.835 us, min = 16.251 us, total = 147.803 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1327 total (0 active), Execution time: mean = 653.737 us, total = 867.509 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 647.697 us, total = 14.249 ms, Queueing time: mean = 84.597 us, max = 153.556 us, min = 48.361 us, total = 1.861 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:48:53,983 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25698 total (0 active) Queueing time: mean = 9.961 us, max = 1.423 ms, min = 568.000 ns, total = 255.967 ms Execution time: mean = 27.052 us, total = 695.180 ms Event stats: event_loop_lag_probe - 25672 total (0 active), Execution time: mean = 27.019 us, total = 693.630 ms, Queueing time: mean = 9.897 us, max = 1.423 ms, min = 1.850 us, total = 254.087 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:49:53,983 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1293 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1339 - cumulative published bytes: 1063912 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:49:53,984 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 78036 total (4 active) Queueing time: mean = 73.880 us, max = 279.303 ms, min = -0.000 s, total = 5.765 s Execution time: mean = 245.556 us, total = 19.162 s Event stats: event_loop_lag_probe - 25912 total (0 active), Execution time: mean = 24.998 us, total = 647.736 ms, Queueing time: mean = 10.012 us, max = 13.097 ms, min = 1.994 us, total = 259.438 ms GcsInMemoryStore.Put - 10408 total (0 active), Execution time: mean = 32.345 us, total = 336.644 ms, Queueing time: mean = 233.829 us, max = 278.545 ms, min = 2.025 us, total = 2.434 s RayletLoadPulled - 6479 total (1 active), Execution time: mean = 249.058 us, total = 1.614 s, Queueing time: mean = 75.424 us, max = 472.090 us, min = -0.000 s, total = 488.672 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6476 total (0 active), Execution time: mean = 77.470 us, total = 501.697 ms, Queueing time: mean = 103.575 us, max = 2.744 ms, min = 11.910 us, total = 670.751 ms NodeManagerService.grpc_client.GetResourceLoad - 6476 total (0 active), Execution time: mean = 1.496 ms, total = 9.685 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2600 total (0 active), Execution time: mean = 43.360 us, total = 112.736 ms, Queueing time: mean = 104.328 us, max = 479.211 us, min = 11.443 us, total = 271.254 ms InternalKVGcsService.grpc_server.InternalKVPut - 2600 total (0 active), Execution time: mean = 569.388 us, total = 1.480 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2591 total (0 active), Execution time: mean = 35.907 us, total = 93.035 ms, Queueing time: mean = 118.446 us, max = 1.498 ms, min = 20.329 us, total = 306.893 ms NodeInfoGcsService.grpc_server.CheckAlive - 2591 total (0 active), Execution time: mean = 596.774 us, total = 1.546 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2161 total (1 active), Execution time: mean = 9.535 us, total = 20.606 ms, Queueing time: mean = 75.657 us, max = 391.368 us, min = 10.687 us, total = 163.496 ms HealthCheck - 2157 total (0 active), Execution time: mean = 7.731 us, total = 16.675 ms, Queueing time: mean = 103.458 us, max = 564.605 us, min = 16.806 us, total = 223.160 ms GcsInMemoryStore.Get - 1362 total (0 active), Execution time: mean = 21.001 us, total = 28.603 ms, Queueing time: mean = 13.490 us, max = 128.508 us, min = 3.701 us, total = 18.373 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1361 total (0 active), Execution time: mean = 58.123 us, total = 79.105 ms, Queueing time: mean = 105.723 us, max = 269.544 us, min = 13.461 us, total = 143.890 ms InternalKVGcsService.grpc_server.InternalKVGet - 1361 total (0 active), Execution time: mean = 613.864 us, total = 835.469 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1293 total (0 active), Execution time: mean = 93.259 us, total = 120.584 ms, Queueing time: mean = 117.684 us, max = 296.469 us, min = 14.122 us, total = 152.166 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1293 total (0 active), Execution time: mean = 685.905 us, total = 886.876 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 648 total (1 active), Execution time: mean = 1.426 ms, total = 924.320 ms, Queueing time: mean = 72.679 us, max = 323.875 us, min = -0.000 s, total = 47.096 ms GCSServer.deadline_timer.debug_state_event_stats_print - 108 total (1 active, 1 running), Execution time: mean = 1.600 ms, total = 172.784 ms, Queueing time: mean = 65.172 us, max = 182.019 us, min = 12.907 us, total = 7.039 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:49:53,984 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 299188 total (1 active) Queueing time: mean = 49.920 us, max = 7.565 ms, min = 1.647 us, total = 14.936 s Execution time: mean = 256.658 us, total = 76.789 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 135990 total (0 active), Execution time: mean = 530.740 us, total = 72.175 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 135990 total (0 active), Execution time: mean = 28.233 us, total = 3.839 s, Queueing time: mean = 107.041 us, max = 7.565 ms, min = 3.372 us, total = 14.556 s event_loop_lag_probe - 25910 total (0 active), Execution time: mean = 29.293 us, total = 758.992 ms, Queueing time: mean = 10.421 us, max = 4.983 ms, min = 1.647 us, total = 270.015 ms GcsTaskManager.GcJobSummary - 1297 total (1 active), Execution time: mean = 11.687 us, total = 15.158 ms, Queueing time: mean = 84.042 us, max = 265.885 us, min = 16.725 us, total = 109.003 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:49:53,984 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 31857 total (25 active) Queueing time: mean = 19.669 us, max = 333.684 us, min = 2.330 us, total = 626.608 ms Execution time: mean = 4.580 s, total = 145901.736 s Event stats: event_loop_lag_probe - 25910 total (0 active), Execution time: mean = 31.875 us, total = 825.870 ms, Queueing time: mean = 11.756 us, max = 291.078 us, min = 2.330 us, total = 304.591 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1598 total (0 active), Execution time: mean = 43.936 us, total = 70.209 ms, Queueing time: mean = 104.933 us, max = 333.684 us, min = 6.593 us, total = 167.683 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1598 total (24 active), Execution time: mean = 91.301 s, total = 145899.777 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1339 total (0 active), Execution time: mean = 117.067 us, total = 156.753 ms, Queueing time: mean = 111.318 us, max = 293.835 us, min = 16.251 us, total = 149.055 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1339 total (0 active), Execution time: mean = 653.445 us, total = 874.962 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 647.697 us, total = 14.249 ms, Queueing time: mean = 84.597 us, max = 153.556 us, min = 48.361 us, total = 1.861 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:49:53,984 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 25938 total (0 active) Queueing time: mean = 9.950 us, max = 1.423 ms, min = 568.000 ns, total = 258.082 ms Execution time: mean = 27.020 us, total = 700.841 ms Event stats: event_loop_lag_probe - 25912 total (0 active), Execution time: mean = 26.987 us, total = 699.292 ms, Queueing time: mean = 9.887 us, max = 1.423 ms, min = 1.850 us, total = 256.203 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:50:53,985 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1305 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1351 - cumulative published bytes: 1073740 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:50:53,986 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 78754 total (4 active) Queueing time: mean = 73.614 us, max = 279.303 ms, min = -0.000 s, total = 5.797 s Execution time: mean = 245.519 us, total = 19.336 s Event stats: event_loop_lag_probe - 26151 total (0 active), Execution time: mean = 25.057 us, total = 655.267 ms, Queueing time: mean = 10.026 us, max = 13.097 ms, min = 1.994 us, total = 262.197 ms GcsInMemoryStore.Put - 10504 total (0 active), Execution time: mean = 32.105 us, total = 337.234 ms, Queueing time: mean = 232.293 us, max = 278.545 ms, min = 2.025 us, total = 2.440 s RayletLoadPulled - 6539 total (1 active), Execution time: mean = 249.048 us, total = 1.629 s, Queueing time: mean = 75.421 us, max = 472.090 us, min = -0.000 s, total = 493.179 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6536 total (0 active), Execution time: mean = 77.449 us, total = 506.204 ms, Queueing time: mean = 103.493 us, max = 2.744 ms, min = 11.910 us, total = 676.433 ms NodeManagerService.grpc_client.GetResourceLoad - 6536 total (0 active), Execution time: mean = 1.495 ms, total = 9.774 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2624 total (0 active), Execution time: mean = 43.331 us, total = 113.700 ms, Queueing time: mean = 104.355 us, max = 479.211 us, min = 11.443 us, total = 273.827 ms InternalKVGcsService.grpc_server.InternalKVPut - 2624 total (0 active), Execution time: mean = 569.467 us, total = 1.494 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2615 total (0 active), Execution time: mean = 35.896 us, total = 93.867 ms, Queueing time: mean = 118.426 us, max = 1.498 ms, min = 20.329 us, total = 309.684 ms NodeInfoGcsService.grpc_server.CheckAlive - 2615 total (0 active), Execution time: mean = 596.675 us, total = 1.560 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2181 total (1 active), Execution time: mean = 9.559 us, total = 20.849 ms, Queueing time: mean = 75.872 us, max = 391.368 us, min = 10.687 us, total = 165.476 ms HealthCheck - 2177 total (0 active), Execution time: mean = 7.734 us, total = 16.836 ms, Queueing time: mean = 103.344 us, max = 564.605 us, min = 16.806 us, total = 224.979 ms GcsInMemoryStore.Get - 1374 total (0 active), Execution time: mean = 21.015 us, total = 28.875 ms, Queueing time: mean = 13.487 us, max = 128.508 us, min = 3.701 us, total = 18.531 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1373 total (0 active), Execution time: mean = 58.153 us, total = 79.845 ms, Queueing time: mean = 105.943 us, max = 269.544 us, min = 13.461 us, total = 145.460 ms InternalKVGcsService.grpc_server.InternalKVGet - 1373 total (0 active), Execution time: mean = 614.182 us, total = 843.272 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1305 total (0 active), Execution time: mean = 93.393 us, total = 121.878 ms, Queueing time: mean = 117.624 us, max = 296.469 us, min = 14.122 us, total = 153.499 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1305 total (0 active), Execution time: mean = 685.955 us, total = 895.172 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 654 total (1 active), Execution time: mean = 1.424 ms, total = 931.216 ms, Queueing time: mean = 72.845 us, max = 323.875 us, min = -0.000 s, total = 47.641 ms GCSServer.deadline_timer.debug_state_event_stats_print - 109 total (1 active, 1 running), Execution time: mean = 1.598 ms, total = 174.230 ms, Queueing time: mean = 64.786 us, max = 182.019 us, min = 12.907 us, total = 7.062 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:50:53,986 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 301960 total (1 active) Queueing time: mean = 49.938 us, max = 7.565 ms, min = 1.647 us, total = 15.079 s Execution time: mean = 256.714 us, total = 77.517 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 137250 total (0 active), Execution time: mean = 530.856 us, total = 72.860 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 137250 total (0 active), Execution time: mean = 28.244 us, total = 3.876 s, Queueing time: mean = 107.082 us, max = 7.565 ms, min = 3.372 us, total = 14.697 s event_loop_lag_probe - 26150 total (0 active), Execution time: mean = 29.277 us, total = 765.606 ms, Queueing time: mean = 10.412 us, max = 4.983 ms, min = 1.647 us, total = 272.263 ms GcsTaskManager.GcJobSummary - 1309 total (1 active), Execution time: mean = 11.696 us, total = 15.310 ms, Queueing time: mean = 84.038 us, max = 265.885 us, min = 16.725 us, total = 110.006 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:50:53,986 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32145 total (25 active) Queueing time: mean = 19.673 us, max = 333.684 us, min = 2.330 us, total = 632.379 ms Execution time: mean = 4.541 s, total = 145961.840 s Event stats: event_loop_lag_probe - 26150 total (0 active), Execution time: mean = 31.887 us, total = 833.849 ms, Queueing time: mean = 11.763 us, max = 291.078 us, min = 2.330 us, total = 307.600 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1610 total (0 active), Execution time: mean = 43.923 us, total = 70.717 ms, Queueing time: mean = 105.005 us, max = 333.684 us, min = 6.593 us, total = 169.058 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1610 total (24 active), Execution time: mean = 90.658 s, total = 145959.864 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1351 total (0 active), Execution time: mean = 116.935 us, total = 157.979 ms, Queueing time: mean = 111.356 us, max = 293.835 us, min = 16.251 us, total = 150.442 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1351 total (0 active), Execution time: mean = 653.455 us, total = 882.817 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 22 total (1 active), Execution time: mean = 647.697 us, total = 14.249 ms, Queueing time: mean = 84.597 us, max = 153.556 us, min = 48.361 us, total = 1.861 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:50:53,986 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26178 total (0 active) Queueing time: mean = 9.959 us, max = 1.423 ms, min = 568.000 ns, total = 260.695 ms Execution time: mean = 27.036 us, total = 707.748 ms Event stats: event_loop_lag_probe - 26152 total (0 active), Execution time: mean = 27.004 us, total = 706.199 ms, Queueing time: mean = 9.897 us, max = 1.423 ms, min = 1.850 us, total = 258.816 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:51:53,986 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1317 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1363 - cumulative published bytes: 1083568 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:51:53,987 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 79473 total (4 active) Queueing time: mean = 73.334 us, max = 279.303 ms, min = -0.000 s, total = 5.828 s Execution time: mean = 245.514 us, total = 19.512 s Event stats: event_loop_lag_probe - 26391 total (0 active), Execution time: mean = 25.077 us, total = 661.816 ms, Queueing time: mean = 10.025 us, max = 13.097 ms, min = 1.994 us, total = 264.575 ms GcsInMemoryStore.Put - 10600 total (0 active), Execution time: mean = 31.869 us, total = 337.813 ms, Queueing time: mean = 230.599 us, max = 278.545 ms, min = 2.025 us, total = 2.444 s RayletLoadPulled - 6599 total (1 active), Execution time: mean = 249.069 us, total = 1.644 s, Queueing time: mean = 75.419 us, max = 472.090 us, min = -0.000 s, total = 497.692 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6596 total (0 active), Execution time: mean = 77.449 us, total = 510.852 ms, Queueing time: mean = 103.539 us, max = 2.744 ms, min = 11.910 us, total = 682.943 ms NodeManagerService.grpc_client.GetResourceLoad - 6596 total (0 active), Execution time: mean = 1.495 ms, total = 9.862 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2648 total (0 active), Execution time: mean = 43.476 us, total = 115.125 ms, Queueing time: mean = 104.465 us, max = 479.211 us, min = 11.443 us, total = 276.624 ms InternalKVGcsService.grpc_server.InternalKVPut - 2648 total (0 active), Execution time: mean = 569.860 us, total = 1.509 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2639 total (0 active), Execution time: mean = 35.976 us, total = 94.941 ms, Queueing time: mean = 118.432 us, max = 1.498 ms, min = 20.329 us, total = 312.541 ms NodeInfoGcsService.grpc_server.CheckAlive - 2639 total (0 active), Execution time: mean = 596.670 us, total = 1.575 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2201 total (1 active), Execution time: mean = 9.564 us, total = 21.050 ms, Queueing time: mean = 75.996 us, max = 391.368 us, min = 10.687 us, total = 167.267 ms HealthCheck - 2197 total (0 active), Execution time: mean = 7.736 us, total = 16.995 ms, Queueing time: mean = 103.325 us, max = 564.605 us, min = 16.806 us, total = 227.004 ms GcsInMemoryStore.Get - 1386 total (0 active), Execution time: mean = 21.007 us, total = 29.115 ms, Queueing time: mean = 13.490 us, max = 128.508 us, min = 3.701 us, total = 18.697 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1385 total (0 active), Execution time: mean = 58.228 us, total = 80.646 ms, Queueing time: mean = 106.057 us, max = 269.544 us, min = 13.461 us, total = 146.889 ms InternalKVGcsService.grpc_server.InternalKVGet - 1385 total (0 active), Execution time: mean = 614.287 us, total = 850.788 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1317 total (0 active), Execution time: mean = 93.532 us, total = 123.182 ms, Queueing time: mean = 117.692 us, max = 296.469 us, min = 14.122 us, total = 155.000 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1317 total (0 active), Execution time: mean = 687.144 us, total = 904.969 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 660 total (1 active), Execution time: mean = 1.422 ms, total = 938.814 ms, Queueing time: mean = 72.665 us, max = 323.875 us, min = -0.000 s, total = 47.959 ms GCSServer.deadline_timer.debug_state_event_stats_print - 110 total (1 active, 1 running), Execution time: mean = 1.599 ms, total = 175.853 ms, Queueing time: mean = 64.804 us, max = 182.019 us, min = 12.907 us, total = 7.128 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:51:53,988 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 304726 total (2 active) Queueing time: mean = 49.955 us, max = 7.565 ms, min = 1.647 us, total = 15.223 s Execution time: mean = 256.816 us, total = 78.259 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 138507 total (1 active), Execution time: mean = 531.043 us, total = 73.553 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 138507 total (0 active), Execution time: mean = 28.282 us, total = 3.917 s, Queueing time: mean = 107.119 us, max = 7.565 ms, min = 3.372 us, total = 14.837 s event_loop_lag_probe - 26390 total (0 active), Execution time: mean = 29.280 us, total = 772.693 ms, Queueing time: mean = 10.411 us, max = 4.983 ms, min = 1.647 us, total = 274.747 ms GcsTaskManager.GcJobSummary - 1321 total (1 active), Execution time: mean = 11.715 us, total = 15.475 ms, Queueing time: mean = 84.052 us, max = 265.885 us, min = 16.725 us, total = 111.032 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:51:53,988 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32470 total (25 active) Queueing time: mean = 19.730 us, max = 333.684 us, min = 2.330 us, total = 640.633 ms Execution time: mean = 4.830 s, total = 156822.065 s Event stats: event_loop_lag_probe - 26390 total (0 active), Execution time: mean = 31.898 us, total = 841.799 ms, Queueing time: mean = 11.766 us, max = 291.078 us, min = 2.330 us, total = 310.518 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1640 total (0 active), Execution time: mean = 43.835 us, total = 71.889 ms, Queueing time: mean = 105.384 us, max = 333.684 us, min = 6.593 us, total = 172.830 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1640 total (24 active), Execution time: mean = 95.622 s, total = 156820.069 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1363 total (0 active), Execution time: mean = 116.957 us, total = 159.413 ms, Queueing time: mean = 111.439 us, max = 293.835 us, min = 16.251 us, total = 151.891 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1363 total (0 active), Execution time: mean = 653.949 us, total = 891.332 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 662.936 us, total = 15.248 ms, Queueing time: mean = 85.961 us, max = 153.556 us, min = 48.361 us, total = 1.977 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:51:53,988 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26418 total (0 active) Queueing time: mean = 9.960 us, max = 1.423 ms, min = 568.000 ns, total = 263.115 ms Execution time: mean = 27.035 us, total = 714.209 ms Event stats: event_loop_lag_probe - 26392 total (0 active), Execution time: mean = 27.003 us, total = 712.660 ms, Queueing time: mean = 9.898 us, max = 1.423 ms, min = 1.850 us, total = 261.236 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:52:53,988 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1329 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1376 - cumulative published bytes: 1093430 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:52:53,989 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 80192 total (4 active) Queueing time: mean = 73.069 us, max = 279.303 ms, min = -0.000 s, total = 5.860 s Execution time: mean = 245.499 us, total = 19.687 s Event stats: event_loop_lag_probe - 26631 total (0 active), Execution time: mean = 25.118 us, total = 668.928 ms, Queueing time: mean = 10.025 us, max = 13.097 ms, min = 1.994 us, total = 266.973 ms GcsInMemoryStore.Put - 10696 total (0 active), Execution time: mean = 31.628 us, total = 338.295 ms, Queueing time: mean = 228.942 us, max = 278.545 ms, min = 2.025 us, total = 2.449 s RayletLoadPulled - 6659 total (1 active), Execution time: mean = 249.022 us, total = 1.658 s, Queueing time: mean = 75.488 us, max = 472.090 us, min = -0.000 s, total = 502.672 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6656 total (0 active), Execution time: mean = 77.451 us, total = 515.515 ms, Queueing time: mean = 103.686 us, max = 2.744 ms, min = 11.910 us, total = 690.131 ms NodeManagerService.grpc_client.GetResourceLoad - 6656 total (0 active), Execution time: mean = 1.495 ms, total = 9.953 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2672 total (0 active), Execution time: mean = 43.498 us, total = 116.227 ms, Queueing time: mean = 104.632 us, max = 479.211 us, min = 11.443 us, total = 279.577 ms InternalKVGcsService.grpc_server.InternalKVPut - 2672 total (0 active), Execution time: mean = 570.067 us, total = 1.523 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2663 total (0 active), Execution time: mean = 36.026 us, total = 95.937 ms, Queueing time: mean = 118.416 us, max = 1.498 ms, min = 20.329 us, total = 315.343 ms NodeInfoGcsService.grpc_server.CheckAlive - 2663 total (0 active), Execution time: mean = 596.376 us, total = 1.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2221 total (1 active), Execution time: mean = 9.562 us, total = 21.237 ms, Queueing time: mean = 75.972 us, max = 391.368 us, min = 10.687 us, total = 168.733 ms HealthCheck - 2217 total (0 active), Execution time: mean = 7.734 us, total = 17.147 ms, Queueing time: mean = 103.237 us, max = 564.605 us, min = 16.806 us, total = 228.876 ms GcsInMemoryStore.Get - 1398 total (0 active), Execution time: mean = 20.988 us, total = 29.341 ms, Queueing time: mean = 13.491 us, max = 128.508 us, min = 3.701 us, total = 18.861 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1397 total (0 active), Execution time: mean = 58.315 us, total = 81.466 ms, Queueing time: mean = 106.059 us, max = 269.544 us, min = 13.461 us, total = 148.164 ms InternalKVGcsService.grpc_server.InternalKVGet - 1397 total (0 active), Execution time: mean = 614.187 us, total = 858.019 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1329 total (0 active), Execution time: mean = 93.617 us, total = 124.416 ms, Queueing time: mean = 117.750 us, max = 296.469 us, min = 14.122 us, total = 156.490 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1329 total (0 active), Execution time: mean = 687.518 us, total = 913.712 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 666 total (1 active), Execution time: mean = 1.421 ms, total = 946.682 ms, Queueing time: mean = 72.678 us, max = 323.875 us, min = -0.000 s, total = 48.404 ms GCSServer.deadline_timer.debug_state_event_stats_print - 111 total (1 active, 1 running), Execution time: mean = 1.599 ms, total = 177.532 ms, Queueing time: mean = 64.875 us, max = 182.019 us, min = 12.907 us, total = 7.201 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:52:53,989 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 307486 total (1 active) Queueing time: mean = 49.978 us, max = 7.565 ms, min = 1.647 us, total = 15.368 s Execution time: mean = 256.880 us, total = 78.987 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 139761 total (0 active), Execution time: mean = 531.165 us, total = 74.236 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 139761 total (0 active), Execution time: mean = 28.299 us, total = 3.955 s, Queueing time: mean = 107.170 us, max = 7.565 ms, min = 3.372 us, total = 14.978 s event_loop_lag_probe - 26630 total (0 active), Execution time: mean = 29.291 us, total = 780.030 ms, Queueing time: mean = 10.415 us, max = 4.983 ms, min = 1.647 us, total = 277.345 ms GcsTaskManager.GcJobSummary - 1333 total (1 active), Execution time: mean = 11.715 us, total = 15.617 ms, Queueing time: mean = 84.018 us, max = 265.885 us, min = 16.725 us, total = 111.996 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:52:53,990 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 32762 total (25 active) Queueing time: mean = 19.729 us, max = 333.684 us, min = 2.330 us, total = 646.348 ms Execution time: mean = 4.789 s, total = 156882.216 s Event stats: event_loop_lag_probe - 26630 total (0 active), Execution time: mean = 31.913 us, total = 849.831 ms, Queueing time: mean = 11.769 us, max = 291.078 us, min = 2.330 us, total = 313.412 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1653 total (0 active), Execution time: mean = 43.879 us, total = 72.532 ms, Queueing time: mean = 105.399 us, max = 333.684 us, min = 6.593 us, total = 174.225 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1653 total (24 active), Execution time: mean = 94.906 s, total = 156880.200 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1376 total (0 active), Execution time: mean = 117.333 us, total = 161.450 ms, Queueing time: mean = 111.422 us, max = 293.835 us, min = 16.251 us, total = 153.316 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1376 total (0 active), Execution time: mean = 654.420 us, total = 900.481 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 662.936 us, total = 15.248 ms, Queueing time: mean = 85.961 us, max = 153.556 us, min = 48.361 us, total = 1.977 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:52:53,990 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26658 total (0 active) Queueing time: mean = 9.956 us, max = 1.423 ms, min = 568.000 ns, total = 265.398 ms Execution time: mean = 27.029 us, total = 720.551 ms Event stats: event_loop_lag_probe - 26632 total (0 active), Execution time: mean = 26.998 us, total = 719.002 ms, Queueing time: mean = 9.895 us, max = 1.423 ms, min = 1.850 us, total = 263.518 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:53:53,990 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1341 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1388 - cumulative published bytes: 1103258 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:53:53,991 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 80911 total (4 active) Queueing time: mean = 72.828 us, max = 279.303 ms, min = -0.000 s, total = 5.893 s Execution time: mean = 245.559 us, total = 19.868 s Event stats: event_loop_lag_probe - 26871 total (0 active), Execution time: mean = 25.141 us, total = 675.575 ms, Queueing time: mean = 10.029 us, max = 13.097 ms, min = 1.994 us, total = 269.497 ms GcsInMemoryStore.Put - 10792 total (0 active), Execution time: mean = 31.402 us, total = 338.891 ms, Queueing time: mean = 227.353 us, max = 278.545 ms, min = 2.025 us, total = 2.454 s RayletLoadPulled - 6719 total (1 active), Execution time: mean = 249.119 us, total = 1.674 s, Queueing time: mean = 75.563 us, max = 472.090 us, min = -0.000 s, total = 507.709 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6716 total (0 active), Execution time: mean = 77.479 us, total = 520.346 ms, Queueing time: mean = 103.786 us, max = 2.744 ms, min = 11.910 us, total = 697.025 ms NodeManagerService.grpc_client.GetResourceLoad - 6716 total (0 active), Execution time: mean = 1.496 ms, total = 10.045 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2696 total (0 active), Execution time: mean = 43.554 us, total = 117.422 ms, Queueing time: mean = 104.748 us, max = 479.211 us, min = 11.443 us, total = 282.402 ms InternalKVGcsService.grpc_server.InternalKVPut - 2696 total (0 active), Execution time: mean = 570.422 us, total = 1.538 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2687 total (0 active), Execution time: mean = 36.074 us, total = 96.930 ms, Queueing time: mean = 118.548 us, max = 1.498 ms, min = 20.329 us, total = 318.539 ms NodeInfoGcsService.grpc_server.CheckAlive - 2687 total (0 active), Execution time: mean = 596.715 us, total = 1.603 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2241 total (1 active), Execution time: mean = 9.563 us, total = 21.431 ms, Queueing time: mean = 76.056 us, max = 391.368 us, min = 10.687 us, total = 170.442 ms HealthCheck - 2237 total (0 active), Execution time: mean = 7.736 us, total = 17.306 ms, Queueing time: mean = 103.346 us, max = 564.605 us, min = 16.806 us, total = 231.186 ms GcsInMemoryStore.Get - 1410 total (0 active), Execution time: mean = 20.995 us, total = 29.603 ms, Queueing time: mean = 13.498 us, max = 128.508 us, min = 3.701 us, total = 19.033 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1409 total (0 active), Execution time: mean = 58.368 us, total = 82.240 ms, Queueing time: mean = 106.155 us, max = 269.544 us, min = 13.461 us, total = 149.572 ms InternalKVGcsService.grpc_server.InternalKVGet - 1409 total (0 active), Execution time: mean = 614.391 us, total = 865.678 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1341 total (0 active), Execution time: mean = 93.646 us, total = 125.579 ms, Queueing time: mean = 117.865 us, max = 296.469 us, min = 14.122 us, total = 158.056 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1341 total (0 active), Execution time: mean = 687.902 us, total = 922.477 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 672 total (1 active), Execution time: mean = 1.422 ms, total = 955.769 ms, Queueing time: mean = 72.748 us, max = 323.875 us, min = -0.000 s, total = 48.887 ms GCSServer.deadline_timer.debug_state_event_stats_print - 112 total (1 active, 1 running), Execution time: mean = 1.601 ms, total = 179.311 ms, Queueing time: mean = 64.859 us, max = 182.019 us, min = 12.907 us, total = 7.264 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:53:53,991 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 310258 total (1 active) Queueing time: mean = 49.989 us, max = 7.565 ms, min = 1.647 us, total = 15.510 s Execution time: mean = 257.002 us, total = 79.737 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 141021 total (0 active), Execution time: mean = 531.406 us, total = 74.939 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 141021 total (0 active), Execution time: mean = 28.324 us, total = 3.994 s, Queueing time: mean = 107.194 us, max = 7.565 ms, min = 3.372 us, total = 15.117 s event_loop_lag_probe - 26870 total (0 active), Execution time: mean = 29.306 us, total = 787.454 ms, Queueing time: mean = 10.419 us, max = 4.983 ms, min = 1.647 us, total = 279.958 ms GcsTaskManager.GcJobSummary - 1345 total (1 active), Execution time: mean = 11.714 us, total = 15.755 ms, Queueing time: mean = 83.950 us, max = 265.885 us, min = 16.725 us, total = 112.912 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:53:53,991 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33050 total (25 active) Queueing time: mean = 19.716 us, max = 333.684 us, min = 2.330 us, total = 651.618 ms Execution time: mean = 4.749 s, total = 156942.446 s Event stats: event_loop_lag_probe - 26870 total (0 active), Execution time: mean = 31.925 us, total = 857.823 ms, Queueing time: mean = 11.775 us, max = 291.078 us, min = 2.330 us, total = 316.382 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1665 total (0 active), Execution time: mean = 43.975 us, total = 73.218 ms, Queueing time: mean = 105.501 us, max = 333.684 us, min = 6.593 us, total = 175.659 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1665 total (24 active), Execution time: mean = 94.259 s, total = 156940.412 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1388 total (0 active), Execution time: mean = 117.219 us, total = 162.699 ms, Queueing time: mean = 111.083 us, max = 293.835 us, min = 16.251 us, total = 154.183 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1388 total (0 active), Execution time: mean = 653.892 us, total = 907.601 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 662.936 us, total = 15.248 ms, Queueing time: mean = 85.961 us, max = 153.556 us, min = 48.361 us, total = 1.977 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:53:53,992 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 26898 total (0 active) Queueing time: mean = 9.957 us, max = 1.423 ms, min = 568.000 ns, total = 267.830 ms Execution time: mean = 27.033 us, total = 727.130 ms Event stats: event_loop_lag_probe - 26872 total (0 active), Execution time: mean = 27.001 us, total = 725.581 ms, Queueing time: mean = 9.897 us, max = 1.423 ms, min = 1.850 us, total = 265.951 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:54:53,992 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1352 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1399 - cumulative published bytes: 1112267 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:54:53,993 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 81619 total (4 active) Queueing time: mean = 72.562 us, max = 279.303 ms, min = -0.000 s, total = 5.922 s Execution time: mean = 245.600 us, total = 20.046 s Event stats: event_loop_lag_probe - 27111 total (0 active), Execution time: mean = 25.149 us, total = 681.823 ms, Queueing time: mean = 10.026 us, max = 13.097 ms, min = 1.994 us, total = 271.808 ms GcsInMemoryStore.Put - 10886 total (0 active), Execution time: mean = 31.173 us, total = 339.350 ms, Queueing time: mean = 225.805 us, max = 278.545 ms, min = 2.025 us, total = 2.458 s RayletLoadPulled - 6779 total (1 active), Execution time: mean = 249.236 us, total = 1.690 s, Queueing time: mean = 75.570 us, max = 472.090 us, min = -0.000 s, total = 512.287 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6776 total (0 active), Execution time: mean = 77.483 us, total = 525.023 ms, Queueing time: mean = 103.761 us, max = 2.744 ms, min = 11.910 us, total = 703.086 ms NodeManagerService.grpc_client.GetResourceLoad - 6776 total (0 active), Execution time: mean = 1.496 ms, total = 10.139 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2718 total (0 active), Execution time: mean = 43.577 us, total = 118.443 ms, Queueing time: mean = 104.845 us, max = 479.211 us, min = 11.443 us, total = 284.968 ms InternalKVGcsService.grpc_server.InternalKVPut - 2718 total (0 active), Execution time: mean = 570.568 us, total = 1.551 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2711 total (0 active), Execution time: mean = 36.064 us, total = 97.771 ms, Queueing time: mean = 118.472 us, max = 1.498 ms, min = 20.329 us, total = 321.176 ms NodeInfoGcsService.grpc_server.CheckAlive - 2711 total (0 active), Execution time: mean = 596.595 us, total = 1.617 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2261 total (1 active), Execution time: mean = 9.555 us, total = 21.603 ms, Queueing time: mean = 76.018 us, max = 391.368 us, min = 10.687 us, total = 171.877 ms HealthCheck - 2257 total (0 active), Execution time: mean = 7.740 us, total = 17.469 ms, Queueing time: mean = 103.518 us, max = 564.605 us, min = 16.806 us, total = 233.640 ms GcsInMemoryStore.Get - 1421 total (0 active), Execution time: mean = 20.993 us, total = 29.831 ms, Queueing time: mean = 13.495 us, max = 128.508 us, min = 3.701 us, total = 19.176 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1420 total (0 active), Execution time: mean = 58.383 us, total = 82.904 ms, Queueing time: mean = 106.118 us, max = 269.544 us, min = 13.461 us, total = 150.687 ms InternalKVGcsService.grpc_server.InternalKVGet - 1420 total (0 active), Execution time: mean = 614.338 us, total = 872.361 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1352 total (0 active), Execution time: mean = 93.821 us, total = 126.845 ms, Queueing time: mean = 118.001 us, max = 296.469 us, min = 14.122 us, total = 159.537 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1352 total (0 active), Execution time: mean = 688.111 us, total = 930.326 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 678 total (1 active), Execution time: mean = 1.422 ms, total = 964.435 ms, Queueing time: mean = 72.795 us, max = 323.875 us, min = -0.000 s, total = 49.355 ms GCSServer.deadline_timer.debug_state_event_stats_print - 113 total (1 active, 1 running), Execution time: mean = 1.602 ms, total = 181.024 ms, Queueing time: mean = 65.108 us, max = 182.019 us, min = 12.907 us, total = 7.357 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:54:53,993 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 313028 total (1 active) Queueing time: mean = 50.007 us, max = 7.565 ms, min = 1.647 us, total = 15.654 s Execution time: mean = 257.089 us, total = 80.476 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 142280 total (0 active), Execution time: mean = 531.571 us, total = 75.632 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 142280 total (0 active), Execution time: mean = 28.351 us, total = 4.034 s, Queueing time: mean = 107.234 us, max = 7.565 ms, min = 3.372 us, total = 15.257 s event_loop_lag_probe - 27110 total (0 active), Execution time: mean = 29.301 us, total = 794.344 ms, Queueing time: mean = 10.416 us, max = 4.983 ms, min = 1.647 us, total = 282.371 ms GcsTaskManager.GcJobSummary - 1357 total (1 active), Execution time: mean = 11.728 us, total = 15.915 ms, Queueing time: mean = 83.864 us, max = 265.885 us, min = 16.725 us, total = 113.803 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:54:53,993 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33334 total (25 active) Queueing time: mean = 19.706 us, max = 333.684 us, min = 2.330 us, total = 656.874 ms Execution time: mean = 4.710 s, total = 156997.649 s Event stats: event_loop_lag_probe - 27110 total (0 active), Execution time: mean = 31.942 us, total = 865.948 ms, Queueing time: mean = 11.782 us, max = 291.078 us, min = 2.330 us, total = 319.406 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1676 total (0 active), Execution time: mean = 44.010 us, total = 73.761 ms, Queueing time: mean = 105.412 us, max = 333.684 us, min = 6.593 us, total = 176.670 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1676 total (24 active), Execution time: mean = 93.673 s, total = 156995.599 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1399 total (0 active), Execution time: mean = 117.183 us, total = 163.938 ms, Queueing time: mean = 111.082 us, max = 293.835 us, min = 16.251 us, total = 155.403 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1399 total (0 active), Execution time: mean = 653.686 us, total = 914.507 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 662.936 us, total = 15.248 ms, Queueing time: mean = 85.961 us, max = 153.556 us, min = 48.361 us, total = 1.977 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:54:53,993 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27138 total (0 active) Queueing time: mean = 9.955 us, max = 1.423 ms, min = 568.000 ns, total = 270.163 ms Execution time: mean = 27.022 us, total = 733.311 ms Event stats: event_loop_lag_probe - 27112 total (0 active), Execution time: mean = 26.990 us, total = 731.762 ms, Queueing time: mean = 9.895 us, max = 1.423 ms, min = 1.850 us, total = 268.283 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:55:53,993 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1364 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1411 - cumulative published bytes: 1122095 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:55:53,994 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 82335 total (4 active) Queueing time: mean = 72.248 us, max = 279.303 ms, min = -0.000 s, total = 5.949 s Execution time: mean = 245.472 us, total = 20.211 s Event stats: event_loop_lag_probe - 27351 total (0 active), Execution time: mean = 25.162 us, total = 688.203 ms, Queueing time: mean = 10.025 us, max = 13.097 ms, min = 1.994 us, total = 274.205 ms GcsInMemoryStore.Put - 10982 total (0 active), Execution time: mean = 30.943 us, total = 339.817 ms, Queueing time: mean = 224.216 us, max = 278.545 ms, min = 2.025 us, total = 2.462 s RayletLoadPulled - 6838 total (1 active), Execution time: mean = 249.107 us, total = 1.703 s, Queueing time: mean = 75.486 us, max = 472.090 us, min = -0.000 s, total = 516.174 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6835 total (0 active), Execution time: mean = 77.428 us, total = 529.221 ms, Queueing time: mean = 103.609 us, max = 2.744 ms, min = 11.910 us, total = 708.166 ms NodeManagerService.grpc_client.GetResourceLoad - 6835 total (0 active), Execution time: mean = 1.496 ms, total = 10.223 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2742 total (0 active), Execution time: mean = 43.484 us, total = 119.234 ms, Queueing time: mean = 104.652 us, max = 479.211 us, min = 11.443 us, total = 286.955 ms InternalKVGcsService.grpc_server.InternalKVPut - 2742 total (0 active), Execution time: mean = 570.286 us, total = 1.564 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2735 total (0 active), Execution time: mean = 36.038 us, total = 98.564 ms, Queueing time: mean = 118.430 us, max = 1.498 ms, min = 20.329 us, total = 323.905 ms NodeInfoGcsService.grpc_server.CheckAlive - 2735 total (0 active), Execution time: mean = 596.377 us, total = 1.631 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2281 total (1 active), Execution time: mean = 9.547 us, total = 21.776 ms, Queueing time: mean = 75.937 us, max = 391.368 us, min = 10.687 us, total = 173.213 ms HealthCheck - 2277 total (0 active), Execution time: mean = 7.731 us, total = 17.603 ms, Queueing time: mean = 103.358 us, max = 564.605 us, min = 16.806 us, total = 235.347 ms GcsInMemoryStore.Get - 1433 total (0 active), Execution time: mean = 21.081 us, total = 30.209 ms, Queueing time: mean = 13.483 us, max = 128.508 us, min = 3.701 us, total = 19.320 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1432 total (0 active), Execution time: mean = 58.457 us, total = 83.711 ms, Queueing time: mean = 105.884 us, max = 269.544 us, min = 13.461 us, total = 151.626 ms InternalKVGcsService.grpc_server.InternalKVGet - 1432 total (0 active), Execution time: mean = 614.250 us, total = 879.605 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1364 total (0 active), Execution time: mean = 93.687 us, total = 127.789 ms, Queueing time: mean = 117.876 us, max = 296.469 us, min = 14.122 us, total = 160.783 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1364 total (0 active), Execution time: mean = 687.555 us, total = 937.825 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 684 total (1 active), Execution time: mean = 1.423 ms, total = 973.167 ms, Queueing time: mean = 72.715 us, max = 323.875 us, min = -0.000 s, total = 49.737 ms GCSServer.deadline_timer.debug_state_event_stats_print - 114 total (1 active, 1 running), Execution time: mean = 1.602 ms, total = 182.621 ms, Queueing time: mean = 65.078 us, max = 182.019 us, min = 12.907 us, total = 7.419 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:55:53,994 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 315792 total (1 active) Queueing time: mean = 50.027 us, max = 7.565 ms, min = 1.647 us, total = 15.798 s Execution time: mean = 257.114 us, total = 81.195 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 143536 total (0 active), Execution time: mean = 531.620 us, total = 76.307 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 143536 total (0 active), Execution time: mean = 28.359 us, total = 4.070 s, Queueing time: mean = 107.279 us, max = 7.565 ms, min = 3.372 us, total = 15.398 s event_loop_lag_probe - 27350 total (0 active), Execution time: mean = 29.310 us, total = 801.620 ms, Queueing time: mean = 10.415 us, max = 4.983 ms, min = 1.647 us, total = 284.861 ms GcsTaskManager.GcJobSummary - 1369 total (1 active), Execution time: mean = 11.719 us, total = 16.043 ms, Queueing time: mean = 83.847 us, max = 265.885 us, min = 16.725 us, total = 114.786 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:55:53,995 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33622 total (25 active) Queueing time: mean = 19.706 us, max = 333.684 us, min = 2.330 us, total = 662.563 ms Execution time: mean = 4.671 s, total = 157057.751 s Event stats: event_loop_lag_probe - 27350 total (0 active), Execution time: mean = 31.953 us, total = 873.904 ms, Queueing time: mean = 11.785 us, max = 291.078 us, min = 2.330 us, total = 322.313 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1688 total (0 active), Execution time: mean = 44.021 us, total = 74.307 ms, Queueing time: mean = 105.487 us, max = 333.684 us, min = 6.593 us, total = 178.061 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1688 total (24 active), Execution time: mean = 93.042 s, total = 157055.683 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1411 total (0 active), Execution time: mean = 117.112 us, total = 165.245 ms, Queueing time: mean = 111.123 us, max = 293.835 us, min = 16.251 us, total = 156.794 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1411 total (0 active), Execution time: mean = 653.438 us, total = 922.002 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 23 total (1 active), Execution time: mean = 662.936 us, total = 15.248 ms, Queueing time: mean = 85.961 us, max = 153.556 us, min = 48.361 us, total = 1.977 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:55:53,995 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27378 total (0 active) Queueing time: mean = 9.953 us, max = 1.423 ms, min = 568.000 ns, total = 272.506 ms Execution time: mean = 27.015 us, total = 739.613 ms Event stats: event_loop_lag_probe - 27352 total (0 active), Execution time: mean = 26.984 us, total = 738.064 ms, Queueing time: mean = 9.894 us, max = 1.423 ms, min = 1.850 us, total = 270.627 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:56:53,995 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1376 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1423 - cumulative published bytes: 1131923 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:56:53,996 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 83054 total (4 active) Queueing time: mean = 71.994 us, max = 279.303 ms, min = -0.000 s, total = 5.979 s Execution time: mean = 245.484 us, total = 20.388 s Event stats: event_loop_lag_probe - 27591 total (0 active), Execution time: mean = 25.201 us, total = 695.330 ms, Queueing time: mean = 10.031 us, max = 13.097 ms, min = 1.994 us, total = 276.776 ms GcsInMemoryStore.Put - 11078 total (0 active), Execution time: mean = 30.721 us, total = 340.332 ms, Queueing time: mean = 222.673 us, max = 278.545 ms, min = 2.025 us, total = 2.467 s RayletLoadPulled - 6898 total (1 active), Execution time: mean = 249.209 us, total = 1.719 s, Queueing time: mean = 75.594 us, max = 472.090 us, min = -0.000 s, total = 521.449 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6895 total (0 active), Execution time: mean = 77.460 us, total = 534.088 ms, Queueing time: mean = 103.658 us, max = 2.744 ms, min = 11.910 us, total = 714.721 ms NodeManagerService.grpc_client.GetResourceLoad - 6895 total (0 active), Execution time: mean = 1.496 ms, total = 10.315 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2766 total (0 active), Execution time: mean = 43.426 us, total = 120.118 ms, Queueing time: mean = 104.552 us, max = 479.211 us, min = 11.443 us, total = 289.192 ms InternalKVGcsService.grpc_server.InternalKVPut - 2766 total (0 active), Execution time: mean = 570.200 us, total = 1.577 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2759 total (0 active), Execution time: mean = 36.072 us, total = 99.524 ms, Queueing time: mean = 118.523 us, max = 1.498 ms, min = 20.329 us, total = 327.006 ms NodeInfoGcsService.grpc_server.CheckAlive - 2759 total (0 active), Execution time: mean = 596.419 us, total = 1.646 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2301 total (1 active), Execution time: mean = 9.552 us, total = 21.978 ms, Queueing time: mean = 75.918 us, max = 391.368 us, min = 10.687 us, total = 174.687 ms HealthCheck - 2297 total (0 active), Execution time: mean = 7.734 us, total = 17.765 ms, Queueing time: mean = 103.399 us, max = 564.605 us, min = 16.806 us, total = 237.507 ms GcsInMemoryStore.Get - 1445 total (0 active), Execution time: mean = 21.084 us, total = 30.466 ms, Queueing time: mean = 13.475 us, max = 128.508 us, min = 3.701 us, total = 19.472 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1444 total (0 active), Execution time: mean = 58.464 us, total = 84.422 ms, Queueing time: mean = 105.747 us, max = 269.544 us, min = 13.461 us, total = 152.699 ms InternalKVGcsService.grpc_server.InternalKVGet - 1444 total (0 active), Execution time: mean = 614.057 us, total = 886.699 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1376 total (0 active), Execution time: mean = 93.691 us, total = 128.918 ms, Queueing time: mean = 117.751 us, max = 296.469 us, min = 14.122 us, total = 162.026 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1376 total (0 active), Execution time: mean = 687.364 us, total = 945.812 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 690 total (1 active), Execution time: mean = 1.424 ms, total = 982.467 ms, Queueing time: mean = 72.770 us, max = 323.875 us, min = -0.000 s, total = 50.211 ms GCSServer.deadline_timer.debug_state_event_stats_print - 115 total (1 active, 1 running), Execution time: mean = 1.601 ms, total = 184.105 ms, Queueing time: mean = 64.978 us, max = 182.019 us, min = 12.907 us, total = 7.472 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:56:53,996 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 318561 total (1 active) Queueing time: mean = 50.046 us, max = 7.565 ms, min = 1.647 us, total = 15.943 s Execution time: mean = 257.172 us, total = 81.925 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 144795 total (0 active), Execution time: mean = 531.728 us, total = 76.992 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 144795 total (0 active), Execution time: mean = 28.367 us, total = 4.107 s, Queueing time: mean = 107.317 us, max = 7.565 ms, min = 3.372 us, total = 15.539 s event_loop_lag_probe - 27589 total (0 active), Execution time: mean = 29.351 us, total = 809.763 ms, Queueing time: mean = 10.428 us, max = 4.983 ms, min = 1.647 us, total = 287.701 ms GcsTaskManager.GcJobSummary - 1381 total (1 active), Execution time: mean = 11.731 us, total = 16.201 ms, Queueing time: mean = 83.899 us, max = 265.885 us, min = 16.725 us, total = 115.865 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:56:53,997 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 33920 total (25 active) Queueing time: mean = 19.718 us, max = 333.684 us, min = 2.330 us, total = 668.846 ms Execution time: mean = 4.720 s, total = 160117.967 s Event stats: event_loop_lag_probe - 27589 total (0 active), Execution time: mean = 31.955 us, total = 881.597 ms, Queueing time: mean = 11.785 us, max = 291.078 us, min = 2.330 us, total = 325.140 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1705 total (0 active), Execution time: mean = 43.984 us, total = 74.993 ms, Queueing time: mean = 105.494 us, max = 333.684 us, min = 6.593 us, total = 179.868 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1705 total (24 active), Execution time: mean = 93.910 s, total = 160115.881 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1423 total (0 active), Execution time: mean = 117.081 us, total = 166.606 ms, Queueing time: mean = 111.258 us, max = 293.835 us, min = 16.251 us, total = 158.320 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1423 total (0 active), Execution time: mean = 653.575 us, total = 930.037 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 642.224 us, total = 15.413 ms, Queueing time: mean = 87.523 us, max = 153.556 us, min = 48.361 us, total = 2.101 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:56:53,997 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27617 total (0 active) Queueing time: mean = 9.954 us, max = 1.423 ms, min = 568.000 ns, total = 274.904 ms Execution time: mean = 27.025 us, total = 746.348 ms Event stats: event_loop_lag_probe - 27591 total (0 active), Execution time: mean = 26.994 us, total = 744.799 ms, Queueing time: mean = 9.895 us, max = 1.423 ms, min = 1.850 us, total = 273.025 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:57:53,997 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1388 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1436 - cumulative published bytes: 1141785 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:57:53,998 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 83773 total (4 active) Queueing time: mean = 71.745 us, max = 279.303 ms, min = -0.000 s, total = 6.010 s Execution time: mean = 245.489 us, total = 20.565 s Event stats: event_loop_lag_probe - 27831 total (0 active), Execution time: mean = 25.217 us, total = 701.808 ms, Queueing time: mean = 10.028 us, max = 13.097 ms, min = 1.994 us, total = 279.099 ms GcsInMemoryStore.Put - 11174 total (0 active), Execution time: mean = 30.502 us, total = 340.835 ms, Queueing time: mean = 221.127 us, max = 278.545 ms, min = 2.025 us, total = 2.471 s RayletLoadPulled - 6958 total (1 active), Execution time: mean = 249.252 us, total = 1.734 s, Queueing time: mean = 75.623 us, max = 472.090 us, min = -0.000 s, total = 526.186 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 6955 total (0 active), Execution time: mean = 77.455 us, total = 538.698 ms, Queueing time: mean = 103.762 us, max = 2.744 ms, min = 11.910 us, total = 721.663 ms NodeManagerService.grpc_client.GetResourceLoad - 6955 total (0 active), Execution time: mean = 1.496 ms, total = 10.407 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2790 total (0 active), Execution time: mean = 43.383 us, total = 121.039 ms, Queueing time: mean = 104.707 us, max = 479.211 us, min = 11.443 us, total = 292.134 ms InternalKVGcsService.grpc_server.InternalKVPut - 2790 total (0 active), Execution time: mean = 570.289 us, total = 1.591 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2783 total (0 active), Execution time: mean = 36.039 us, total = 100.297 ms, Queueing time: mean = 118.380 us, max = 1.498 ms, min = 20.329 us, total = 329.451 ms NodeInfoGcsService.grpc_server.CheckAlive - 2783 total (0 active), Execution time: mean = 596.333 us, total = 1.660 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2321 total (1 active), Execution time: mean = 9.545 us, total = 22.153 ms, Queueing time: mean = 75.874 us, max = 391.368 us, min = 10.687 us, total = 176.104 ms HealthCheck - 2317 total (0 active), Execution time: mean = 7.733 us, total = 17.917 ms, Queueing time: mean = 103.570 us, max = 564.605 us, min = 16.806 us, total = 239.971 ms GcsInMemoryStore.Get - 1457 total (0 active), Execution time: mean = 21.095 us, total = 30.736 ms, Queueing time: mean = 13.489 us, max = 128.508 us, min = 3.701 us, total = 19.654 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1456 total (0 active), Execution time: mean = 58.463 us, total = 85.122 ms, Queueing time: mean = 105.800 us, max = 269.544 us, min = 13.461 us, total = 154.045 ms InternalKVGcsService.grpc_server.InternalKVGet - 1456 total (0 active), Execution time: mean = 614.310 us, total = 894.435 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1388 total (0 active), Execution time: mean = 93.593 us, total = 129.908 ms, Queueing time: mean = 117.808 us, max = 296.469 us, min = 14.122 us, total = 163.517 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1388 total (0 active), Execution time: mean = 687.316 us, total = 953.995 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 696 total (1 active), Execution time: mean = 1.423 ms, total = 990.189 ms, Queueing time: mean = 72.836 us, max = 323.875 us, min = -0.000 s, total = 50.694 ms GCSServer.deadline_timer.debug_state_event_stats_print - 116 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 185.950 ms, Queueing time: mean = 65.018 us, max = 182.019 us, min = 12.907 us, total = 7.542 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:57:53,998 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 321333 total (1 active) Queueing time: mean = 50.073 us, max = 7.565 ms, min = 1.647 us, total = 16.090 s Execution time: mean = 257.246 us, total = 82.662 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 146055 total (0 active), Execution time: mean = 531.870 us, total = 77.682 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 146055 total (0 active), Execution time: mean = 28.391 us, total = 4.147 s, Queueing time: mean = 107.377 us, max = 7.565 ms, min = 3.372 us, total = 15.683 s event_loop_lag_probe - 27829 total (0 active), Execution time: mean = 29.335 us, total = 816.363 ms, Queueing time: mean = 10.423 us, max = 4.983 ms, min = 1.647 us, total = 290.075 ms GcsTaskManager.GcJobSummary - 1393 total (1 active), Execution time: mean = 11.734 us, total = 16.345 ms, Queueing time: mean = 83.859 us, max = 265.885 us, min = 16.725 us, total = 116.815 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:57:53,998 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34212 total (25 active) Queueing time: mean = 19.720 us, max = 333.684 us, min = 2.330 us, total = 674.675 ms Execution time: mean = 4.682 s, total = 160178.178 s Event stats: event_loop_lag_probe - 27829 total (0 active), Execution time: mean = 31.959 us, total = 889.389 ms, Queueing time: mean = 11.783 us, max = 291.078 us, min = 2.330 us, total = 327.915 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1718 total (0 active), Execution time: mean = 43.996 us, total = 75.585 ms, Queueing time: mean = 105.568 us, max = 333.684 us, min = 6.593 us, total = 181.366 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1718 total (24 active), Execution time: mean = 93.234 s, total = 160176.074 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1436 total (0 active), Execution time: mean = 117.231 us, total = 168.344 ms, Queueing time: mean = 111.335 us, max = 293.835 us, min = 16.251 us, total = 159.877 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1436 total (0 active), Execution time: mean = 653.737 us, total = 938.766 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 642.224 us, total = 15.413 ms, Queueing time: mean = 87.523 us, max = 153.556 us, min = 48.361 us, total = 2.101 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:57:53,998 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 27857 total (0 active) Queueing time: mean = 9.948 us, max = 1.423 ms, min = 568.000 ns, total = 277.135 ms Execution time: mean = 27.011 us, total = 752.432 ms Event stats: event_loop_lag_probe - 27831 total (0 active), Execution time: mean = 26.980 us, total = 750.883 ms, Queueing time: mean = 9.890 us, max = 1.423 ms, min = 1.850 us, total = 275.256 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:58:53,999 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1400 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1449 - cumulative published bytes: 1151647 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:58:54,000 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 84492 total (4 active) Queueing time: mean = 71.504 us, max = 279.303 ms, min = -0.000 s, total = 6.041 s Execution time: mean = 245.498 us, total = 20.743 s Event stats: event_loop_lag_probe - 28071 total (0 active), Execution time: mean = 25.253 us, total = 708.878 ms, Queueing time: mean = 10.031 us, max = 13.097 ms, min = 1.994 us, total = 281.592 ms GcsInMemoryStore.Put - 11270 total (0 active), Execution time: mean = 30.284 us, total = 341.305 ms, Queueing time: mean = 219.636 us, max = 278.545 ms, min = 2.025 us, total = 2.475 s RayletLoadPulled - 7018 total (1 active), Execution time: mean = 249.265 us, total = 1.749 s, Queueing time: mean = 75.648 us, max = 472.090 us, min = -0.000 s, total = 530.899 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7015 total (0 active), Execution time: mean = 77.451 us, total = 543.319 ms, Queueing time: mean = 103.901 us, max = 2.744 ms, min = 11.910 us, total = 728.864 ms NodeManagerService.grpc_client.GetResourceLoad - 7015 total (0 active), Execution time: mean = 1.497 ms, total = 10.498 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2814 total (0 active), Execution time: mean = 43.384 us, total = 122.083 ms, Queueing time: mean = 104.710 us, max = 479.211 us, min = 11.443 us, total = 294.654 ms InternalKVGcsService.grpc_server.InternalKVPut - 2814 total (0 active), Execution time: mean = 570.324 us, total = 1.605 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2807 total (0 active), Execution time: mean = 36.024 us, total = 101.119 ms, Queueing time: mean = 118.260 us, max = 1.498 ms, min = 20.329 us, total = 331.957 ms NodeInfoGcsService.grpc_server.CheckAlive - 2807 total (0 active), Execution time: mean = 596.159 us, total = 1.673 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2341 total (1 active), Execution time: mean = 9.552 us, total = 22.362 ms, Queueing time: mean = 75.893 us, max = 391.368 us, min = 10.687 us, total = 177.665 ms HealthCheck - 2337 total (0 active), Execution time: mean = 7.733 us, total = 18.072 ms, Queueing time: mean = 103.650 us, max = 564.605 us, min = 16.806 us, total = 242.230 ms GcsInMemoryStore.Get - 1469 total (0 active), Execution time: mean = 21.084 us, total = 30.973 ms, Queueing time: mean = 13.495 us, max = 128.508 us, min = 3.701 us, total = 19.824 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1468 total (0 active), Execution time: mean = 58.504 us, total = 85.884 ms, Queueing time: mean = 105.826 us, max = 269.544 us, min = 13.461 us, total = 155.353 ms InternalKVGcsService.grpc_server.InternalKVGet - 1468 total (0 active), Execution time: mean = 614.262 us, total = 901.737 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1400 total (0 active), Execution time: mean = 93.705 us, total = 131.187 ms, Queueing time: mean = 117.761 us, max = 296.469 us, min = 14.122 us, total = 164.865 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1400 total (0 active), Execution time: mean = 687.030 us, total = 961.842 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 702 total (1 active), Execution time: mean = 1.425 ms, total = 1.001 s, Queueing time: mean = 73.055 us, max = 323.875 us, min = -0.000 s, total = 51.284 ms GCSServer.deadline_timer.debug_state_event_stats_print - 117 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 187.582 ms, Queueing time: mean = 65.026 us, max = 182.019 us, min = 12.907 us, total = 7.608 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:58:54,000 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 324105 total (1 active) Queueing time: mean = 50.091 us, max = 7.565 ms, min = 1.647 us, total = 16.235 s Execution time: mean = 257.332 us, total = 83.403 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 147315 total (0 active), Execution time: mean = 532.035 us, total = 78.377 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 147315 total (0 active), Execution time: mean = 28.418 us, total = 4.186 s, Queueing time: mean = 107.417 us, max = 7.565 ms, min = 3.372 us, total = 15.824 s event_loop_lag_probe - 28069 total (0 active), Execution time: mean = 29.322 us, total = 823.034 ms, Queueing time: mean = 10.418 us, max = 4.983 ms, min = 1.647 us, total = 292.412 ms GcsTaskManager.GcJobSummary - 1405 total (1 active), Execution time: mean = 11.749 us, total = 16.507 ms, Queueing time: mean = 83.932 us, max = 265.885 us, min = 16.725 us, total = 117.924 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:58:54,000 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34504 total (25 active) Queueing time: mean = 19.719 us, max = 333.684 us, min = 2.330 us, total = 680.377 ms Execution time: mean = 4.644 s, total = 160238.360 s Event stats: event_loop_lag_probe - 28069 total (0 active), Execution time: mean = 31.956 us, total = 896.979 ms, Queueing time: mean = 11.781 us, max = 291.078 us, min = 2.330 us, total = 330.674 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1731 total (0 active), Execution time: mean = 44.051 us, total = 76.252 ms, Queueing time: mean = 105.608 us, max = 333.684 us, min = 6.593 us, total = 182.807 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1731 total (24 active), Execution time: mean = 92.569 s, total = 160236.237 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1449 total (0 active), Execution time: mean = 117.183 us, total = 169.798 ms, Queueing time: mean = 111.372 us, max = 293.835 us, min = 16.251 us, total = 161.378 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1449 total (0 active), Execution time: mean = 653.868 us, total = 947.454 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 642.224 us, total = 15.413 ms, Queueing time: mean = 87.523 us, max = 153.556 us, min = 48.361 us, total = 2.101 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:58:54,000 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28097 total (0 active) Queueing time: mean = 9.945 us, max = 1.423 ms, min = 568.000 ns, total = 279.419 ms Execution time: mean = 27.005 us, total = 758.769 ms Event stats: event_loop_lag_probe - 28071 total (0 active), Execution time: mean = 26.975 us, total = 757.220 ms, Queueing time: mean = 9.887 us, max = 1.423 ms, min = 1.850 us, total = 277.540 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 00:59:54,000 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1412 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1461 - cumulative published bytes: 1161475 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 00:59:54,001 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 85210 total (4 active) Queueing time: mean = 71.256 us, max = 279.303 ms, min = -0.000 s, total = 6.072 s Execution time: mean = 245.377 us, total = 20.909 s Event stats: event_loop_lag_probe - 28310 total (0 active), Execution time: mean = 25.277 us, total = 715.596 ms, Queueing time: mean = 10.036 us, max = 13.097 ms, min = 1.994 us, total = 284.117 ms GcsInMemoryStore.Put - 11366 total (0 active), Execution time: mean = 30.076 us, total = 341.841 ms, Queueing time: mean = 218.248 us, max = 278.545 ms, min = 2.025 us, total = 2.481 s RayletLoadPulled - 7078 total (1 active), Execution time: mean = 249.109 us, total = 1.763 s, Queueing time: mean = 75.648 us, max = 472.090 us, min = -0.000 s, total = 535.438 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7075 total (0 active), Execution time: mean = 77.415 us, total = 547.710 ms, Queueing time: mean = 103.850 us, max = 2.744 ms, min = 11.910 us, total = 734.739 ms NodeManagerService.grpc_client.GetResourceLoad - 7075 total (0 active), Execution time: mean = 1.496 ms, total = 10.582 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2838 total (0 active), Execution time: mean = 43.357 us, total = 123.046 ms, Queueing time: mean = 104.689 us, max = 479.211 us, min = 11.443 us, total = 297.107 ms InternalKVGcsService.grpc_server.InternalKVPut - 2838 total (0 active), Execution time: mean = 570.051 us, total = 1.618 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2831 total (0 active), Execution time: mean = 36.025 us, total = 101.986 ms, Queueing time: mean = 118.114 us, max = 1.498 ms, min = 20.329 us, total = 334.380 ms NodeInfoGcsService.grpc_server.CheckAlive - 2831 total (0 active), Execution time: mean = 596.007 us, total = 1.687 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2361 total (1 active), Execution time: mean = 9.559 us, total = 22.568 ms, Queueing time: mean = 75.882 us, max = 391.368 us, min = 10.687 us, total = 179.158 ms HealthCheck - 2357 total (0 active), Execution time: mean = 7.732 us, total = 18.223 ms, Queueing time: mean = 103.716 us, max = 564.605 us, min = 16.806 us, total = 244.458 ms GcsInMemoryStore.Get - 1481 total (0 active), Execution time: mean = 21.087 us, total = 31.230 ms, Queueing time: mean = 13.494 us, max = 128.508 us, min = 3.701 us, total = 19.984 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1480 total (0 active), Execution time: mean = 58.514 us, total = 86.601 ms, Queueing time: mean = 105.891 us, max = 269.544 us, min = 13.461 us, total = 156.718 ms InternalKVGcsService.grpc_server.InternalKVGet - 1480 total (0 active), Execution time: mean = 614.226 us, total = 909.055 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1412 total (0 active), Execution time: mean = 93.674 us, total = 132.268 ms, Queueing time: mean = 117.693 us, max = 296.469 us, min = 14.122 us, total = 166.182 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1412 total (0 active), Execution time: mean = 686.803 us, total = 969.766 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 708 total (1 active), Execution time: mean = 1.425 ms, total = 1.009 s, Queueing time: mean = 73.111 us, max = 323.875 us, min = -0.000 s, total = 51.762 ms GCSServer.deadline_timer.debug_state_event_stats_print - 118 total (1 active, 1 running), Execution time: mean = 1.605 ms, total = 189.351 ms, Queueing time: mean = 65.321 us, max = 182.019 us, min = 12.907 us, total = 7.708 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 00:59:54,002 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 326877 total (1 active) Queueing time: mean = 50.099 us, max = 7.565 ms, min = 1.647 us, total = 16.376 s Execution time: mean = 257.302 us, total = 84.106 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 148575 total (0 active), Execution time: mean = 531.969 us, total = 79.037 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 148575 total (0 active), Execution time: mean = 28.423 us, total = 4.223 s, Queueing time: mean = 107.436 us, max = 7.565 ms, min = 3.372 us, total = 15.962 s event_loop_lag_probe - 28309 total (0 active), Execution time: mean = 29.290 us, total = 829.161 ms, Queueing time: mean = 10.406 us, max = 4.983 ms, min = 1.647 us, total = 294.584 ms GcsTaskManager.GcJobSummary - 1417 total (1 active), Execution time: mean = 11.739 us, total = 16.634 ms, Queueing time: mean = 84.063 us, max = 338.131 us, min = 16.725 us, total = 119.117 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 00:59:54,002 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 34792 total (25 active) Queueing time: mean = 19.711 us, max = 333.684 us, min = 2.330 us, total = 685.791 ms Execution time: mean = 4.607 s, total = 160298.493 s Event stats: event_loop_lag_probe - 28309 total (0 active), Execution time: mean = 31.943 us, total = 904.266 ms, Queueing time: mean = 11.776 us, max = 291.078 us, min = 2.330 us, total = 333.355 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1743 total (0 active), Execution time: mean = 44.085 us, total = 76.840 ms, Queueing time: mean = 105.690 us, max = 333.684 us, min = 6.593 us, total = 184.218 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1743 total (24 active), Execution time: mean = 91.966 s, total = 160296.353 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1461 total (0 active), Execution time: mean = 117.172 us, total = 171.188 ms, Queueing time: mean = 111.362 us, max = 293.835 us, min = 16.251 us, total = 162.700 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1461 total (0 active), Execution time: mean = 653.560 us, total = 954.852 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 642.224 us, total = 15.413 ms, Queueing time: mean = 87.523 us, max = 153.556 us, min = 48.361 us, total = 2.101 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 00:59:54,002 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28337 total (0 active) Queueing time: mean = 9.943 us, max = 1.423 ms, min = 568.000 ns, total = 281.743 ms Execution time: mean = 27.004 us, total = 765.219 ms Event stats: event_loop_lag_probe - 28311 total (0 active), Execution time: mean = 26.974 us, total = 763.670 ms, Queueing time: mean = 9.885 us, max = 1.423 ms, min = 1.850 us, total = 279.864 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:00:54,002 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1424 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1473 - cumulative published bytes: 1171303 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:00:54,003 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 85929 total (4 active) Queueing time: mean = 70.924 us, max = 279.303 ms, min = -0.000 s, total = 6.094 s Execution time: mean = 245.108 us, total = 21.062 s Event stats: event_loop_lag_probe - 28550 total (0 active), Execution time: mean = 25.267 us, total = 721.370 ms, Queueing time: mean = 10.027 us, max = 13.097 ms, min = 1.994 us, total = 286.279 ms GcsInMemoryStore.Put - 11462 total (0 active), Execution time: mean = 29.864 us, total = 342.299 ms, Queueing time: mean = 216.762 us, max = 278.545 ms, min = 1.649 us, total = 2.485 s RayletLoadPulled - 7138 total (1 active), Execution time: mean = 248.855 us, total = 1.776 s, Queueing time: mean = 75.490 us, max = 472.090 us, min = -0.000 s, total = 538.851 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7135 total (0 active), Execution time: mean = 77.346 us, total = 551.866 ms, Queueing time: mean = 103.542 us, max = 2.744 ms, min = 11.645 us, total = 738.770 ms NodeManagerService.grpc_client.GetResourceLoad - 7135 total (0 active), Execution time: mean = 1.494 ms, total = 10.660 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2862 total (0 active), Execution time: mean = 43.408 us, total = 124.234 ms, Queueing time: mean = 104.492 us, max = 479.211 us, min = 11.443 us, total = 299.056 ms InternalKVGcsService.grpc_server.InternalKVPut - 2862 total (0 active), Execution time: mean = 569.695 us, total = 1.630 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2855 total (0 active), Execution time: mean = 36.055 us, total = 102.936 ms, Queueing time: mean = 117.750 us, max = 1.498 ms, min = 20.329 us, total = 336.177 ms NodeInfoGcsService.grpc_server.CheckAlive - 2855 total (0 active), Execution time: mean = 595.319 us, total = 1.700 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2381 total (1 active), Execution time: mean = 9.559 us, total = 22.759 ms, Queueing time: mean = 75.824 us, max = 391.368 us, min = 10.687 us, total = 180.536 ms HealthCheck - 2377 total (0 active), Execution time: mean = 7.719 us, total = 18.349 ms, Queueing time: mean = 103.391 us, max = 564.605 us, min = 16.806 us, total = 245.760 ms GcsInMemoryStore.Get - 1493 total (0 active), Execution time: mean = 21.063 us, total = 31.448 ms, Queueing time: mean = 13.477 us, max = 128.508 us, min = 3.701 us, total = 20.122 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1492 total (0 active), Execution time: mean = 58.471 us, total = 87.239 ms, Queueing time: mean = 105.715 us, max = 269.544 us, min = 13.461 us, total = 157.727 ms InternalKVGcsService.grpc_server.InternalKVGet - 1492 total (0 active), Execution time: mean = 613.582 us, total = 915.465 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1424 total (0 active), Execution time: mean = 93.707 us, total = 133.439 ms, Queueing time: mean = 117.553 us, max = 296.469 us, min = 14.122 us, total = 167.396 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1424 total (0 active), Execution time: mean = 686.227 us, total = 977.187 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 714 total (1 active), Execution time: mean = 1.423 ms, total = 1.016 s, Queueing time: mean = 72.941 us, max = 323.875 us, min = -0.000 s, total = 52.080 ms GCSServer.deadline_timer.debug_state_event_stats_print - 119 total (1 active, 1 running), Execution time: mean = 1.605 ms, total = 191.005 ms, Queueing time: mean = 65.370 us, max = 182.019 us, min = 12.907 us, total = 7.779 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:00:54,003 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 329649 total (1 active) Queueing time: mean = 50.086 us, max = 7.565 ms, min = 1.647 us, total = 16.511 s Execution time: mean = 257.196 us, total = 84.784 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 149835 total (0 active), Execution time: mean = 531.747 us, total = 79.674 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 149835 total (0 active), Execution time: mean = 28.422 us, total = 4.259 s, Queueing time: mean = 107.411 us, max = 7.565 ms, min = 3.372 us, total = 16.094 s event_loop_lag_probe - 28549 total (0 active), Execution time: mean = 29.240 us, total = 834.775 ms, Queueing time: mean = 10.388 us, max = 4.983 ms, min = 1.647 us, total = 296.572 ms GcsTaskManager.GcJobSummary - 1429 total (1 active), Execution time: mean = 11.738 us, total = 16.774 ms, Queueing time: mean = 84.031 us, max = 338.131 us, min = 16.725 us, total = 120.081 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:00:54,003 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35080 total (25 active) Queueing time: mean = 19.696 us, max = 333.684 us, min = 2.330 us, total = 690.936 ms Execution time: mean = 4.571 s, total = 160358.675 s Event stats: event_loop_lag_probe - 28549 total (0 active), Execution time: mean = 31.913 us, total = 911.096 ms, Queueing time: mean = 11.763 us, max = 291.078 us, min = 2.330 us, total = 335.827 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1755 total (0 active), Execution time: mean = 44.078 us, total = 77.357 ms, Queueing time: mean = 105.766 us, max = 333.684 us, min = 6.593 us, total = 185.619 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1755 total (24 active), Execution time: mean = 91.371 s, total = 160356.519 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1473 total (0 active), Execution time: mean = 117.190 us, total = 172.620 ms, Queueing time: mean = 111.318 us, max = 293.835 us, min = 16.251 us, total = 163.972 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1473 total (0 active), Execution time: mean = 653.440 us, total = 962.518 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms Publisher.CheckDeadSubscribers - 24 total (1 active), Execution time: mean = 642.224 us, total = 15.413 ms, Queueing time: mean = 87.523 us, max = 153.556 us, min = 48.361 us, total = 2.101 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:00:54,004 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28577 total (0 active) Queueing time: mean = 9.937 us, max = 1.423 ms, min = 568.000 ns, total = 283.982 ms Execution time: mean = 26.993 us, total = 771.393 ms Event stats: event_loop_lag_probe - 28551 total (0 active), Execution time: mean = 26.964 us, total = 769.844 ms, Queueing time: mean = 9.881 us, max = 1.423 ms, min = 1.850 us, total = 282.102 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:01:54,004 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1436 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1485 - cumulative published bytes: 1181131 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:01:54,005 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 86648 total (4 active) Queueing time: mean = 70.699 us, max = 279.303 ms, min = -0.000 s, total = 6.126 s Execution time: mean = 245.103 us, total = 21.238 s Event stats: event_loop_lag_probe - 28790 total (0 active), Execution time: mean = 25.290 us, total = 728.088 ms, Queueing time: mean = 10.031 us, max = 13.097 ms, min = 1.994 us, total = 288.795 ms GcsInMemoryStore.Put - 11558 total (0 active), Execution time: mean = 29.659 us, total = 342.797 ms, Queueing time: mean = 215.315 us, max = 278.545 ms, min = 1.649 us, total = 2.489 s RayletLoadPulled - 7198 total (1 active), Execution time: mean = 248.937 us, total = 1.792 s, Queueing time: mean = 75.500 us, max = 472.090 us, min = -0.000 s, total = 543.450 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7195 total (0 active), Execution time: mean = 77.348 us, total = 556.517 ms, Queueing time: mean = 103.644 us, max = 2.744 ms, min = 11.645 us, total = 745.721 ms NodeManagerService.grpc_client.GetResourceLoad - 7195 total (0 active), Execution time: mean = 1.494 ms, total = 10.750 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2886 total (0 active), Execution time: mean = 43.364 us, total = 125.148 ms, Queueing time: mean = 104.579 us, max = 479.211 us, min = 11.443 us, total = 301.815 ms InternalKVGcsService.grpc_server.InternalKVPut - 2886 total (0 active), Execution time: mean = 569.711 us, total = 1.644 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2879 total (0 active), Execution time: mean = 36.057 us, total = 103.807 ms, Queueing time: mean = 117.822 us, max = 1.498 ms, min = 20.329 us, total = 339.211 ms NodeInfoGcsService.grpc_server.CheckAlive - 2879 total (0 active), Execution time: mean = 595.691 us, total = 1.715 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2401 total (1 active), Execution time: mean = 9.573 us, total = 22.984 ms, Queueing time: mean = 75.801 us, max = 391.368 us, min = 10.687 us, total = 181.998 ms HealthCheck - 2397 total (0 active), Execution time: mean = 7.724 us, total = 18.515 ms, Queueing time: mean = 103.534 us, max = 564.605 us, min = 16.806 us, total = 248.171 ms GcsInMemoryStore.Get - 1505 total (0 active), Execution time: mean = 21.051 us, total = 31.682 ms, Queueing time: mean = 13.541 us, max = 128.508 us, min = 3.701 us, total = 20.379 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1504 total (0 active), Execution time: mean = 58.457 us, total = 87.919 ms, Queueing time: mean = 105.775 us, max = 269.544 us, min = 13.461 us, total = 159.086 ms InternalKVGcsService.grpc_server.InternalKVGet - 1504 total (0 active), Execution time: mean = 613.616 us, total = 922.879 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1436 total (0 active), Execution time: mean = 93.594 us, total = 134.401 ms, Queueing time: mean = 117.635 us, max = 296.469 us, min = 14.122 us, total = 168.924 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1436 total (0 active), Execution time: mean = 685.826 us, total = 984.846 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 720 total (1 active), Execution time: mean = 1.422 ms, total = 1.024 s, Queueing time: mean = 72.953 us, max = 323.875 us, min = -0.000 s, total = 52.526 ms GCSServer.deadline_timer.debug_state_event_stats_print - 120 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 192.536 ms, Queueing time: mean = 65.305 us, max = 182.019 us, min = 12.907 us, total = 7.837 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:01:54,005 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 332421 total (1 active) Queueing time: mean = 50.090 us, max = 7.565 ms, min = 1.647 us, total = 16.651 s Execution time: mean = 257.280 us, total = 85.525 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 151095 total (0 active), Execution time: mean = 531.907 us, total = 80.369 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 151095 total (0 active), Execution time: mean = 28.449 us, total = 4.298 s, Queueing time: mean = 107.422 us, max = 7.565 ms, min = 3.372 us, total = 16.231 s event_loop_lag_probe - 28789 total (0 active), Execution time: mean = 29.228 us, total = 841.440 ms, Queueing time: mean = 10.385 us, max = 4.983 ms, min = 1.647 us, total = 298.970 ms GcsTaskManager.GcJobSummary - 1441 total (1 active), Execution time: mean = 11.728 us, total = 16.900 ms, Queueing time: mean = 83.914 us, max = 338.131 us, min = 16.725 us, total = 120.921 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:01:54,005 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35405 total (25 active) Queueing time: mean = 19.713 us, max = 333.684 us, min = 2.330 us, total = 697.929 ms Execution time: mean = 4.836 s, total = 171218.830 s Event stats: event_loop_lag_probe - 28789 total (0 active), Execution time: mean = 31.910 us, total = 918.653 ms, Queueing time: mean = 11.761 us, max = 291.078 us, min = 2.330 us, total = 338.596 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1785 total (0 active), Execution time: mean = 43.841 us, total = 78.257 ms, Queueing time: mean = 105.622 us, max = 333.684 us, min = 6.593 us, total = 188.536 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1785 total (24 active), Execution time: mean = 95.920 s, total = 171216.656 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1485 total (0 active), Execution time: mean = 117.154 us, total = 173.973 ms, Queueing time: mean = 111.232 us, max = 293.835 us, min = 16.251 us, total = 165.180 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1485 total (0 active), Execution time: mean = 653.322 us, total = 970.183 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 651.471 us, total = 16.287 ms, Queueing time: mean = 87.988 us, max = 153.556 us, min = 48.361 us, total = 2.200 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:01:54,005 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 28817 total (0 active) Queueing time: mean = 9.937 us, max = 1.423 ms, min = 568.000 ns, total = 286.367 ms Execution time: mean = 26.997 us, total = 777.972 ms Event stats: event_loop_lag_probe - 28791 total (0 active), Execution time: mean = 26.968 us, total = 776.423 ms, Queueing time: mean = 9.881 us, max = 1.423 ms, min = 1.850 us, total = 284.488 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:02:54,005 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1448 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1497 - cumulative published bytes: 1190959 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:02:54,006 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 87367 total (4 active) Queueing time: mean = 70.443 us, max = 279.303 ms, min = -0.000 s, total = 6.154 s Execution time: mean = 245.039 us, total = 21.408 s Event stats: event_loop_lag_probe - 29030 total (0 active), Execution time: mean = 25.306 us, total = 734.647 ms, Queueing time: mean = 10.034 us, max = 13.097 ms, min = 1.994 us, total = 291.275 ms GcsInMemoryStore.Put - 11654 total (0 active), Execution time: mean = 29.455 us, total = 343.263 ms, Queueing time: mean = 213.947 us, max = 278.545 ms, min = 1.649 us, total = 2.493 s RayletLoadPulled - 7258 total (1 active), Execution time: mean = 248.874 us, total = 1.806 s, Queueing time: mean = 75.487 us, max = 472.090 us, min = -0.000 s, total = 547.883 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7255 total (0 active), Execution time: mean = 77.318 us, total = 560.946 ms, Queueing time: mean = 103.536 us, max = 2.744 ms, min = 11.645 us, total = 751.157 ms NodeManagerService.grpc_client.GetResourceLoad - 7255 total (0 active), Execution time: mean = 1.494 ms, total = 10.839 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2910 total (0 active), Execution time: mean = 43.310 us, total = 126.031 ms, Queueing time: mean = 104.503 us, max = 479.211 us, min = 11.443 us, total = 304.104 ms InternalKVGcsService.grpc_server.InternalKVPut - 2910 total (0 active), Execution time: mean = 569.544 us, total = 1.657 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2903 total (0 active), Execution time: mean = 36.074 us, total = 104.723 ms, Queueing time: mean = 117.654 us, max = 1.498 ms, min = 20.329 us, total = 341.551 ms NodeInfoGcsService.grpc_server.CheckAlive - 2903 total (0 active), Execution time: mean = 595.576 us, total = 1.729 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2421 total (1 active), Execution time: mean = 9.588 us, total = 23.213 ms, Queueing time: mean = 75.832 us, max = 391.368 us, min = 10.687 us, total = 183.589 ms HealthCheck - 2417 total (0 active), Execution time: mean = 7.719 us, total = 18.656 ms, Queueing time: mean = 103.457 us, max = 564.605 us, min = 16.806 us, total = 250.055 ms GcsInMemoryStore.Get - 1517 total (0 active), Execution time: mean = 21.029 us, total = 31.901 ms, Queueing time: mean = 13.533 us, max = 128.508 us, min = 3.701 us, total = 20.529 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1516 total (0 active), Execution time: mean = 58.416 us, total = 88.559 ms, Queueing time: mean = 105.794 us, max = 269.544 us, min = 13.461 us, total = 160.383 ms InternalKVGcsService.grpc_server.InternalKVGet - 1516 total (0 active), Execution time: mean = 613.681 us, total = 930.340 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1448 total (0 active), Execution time: mean = 93.665 us, total = 135.627 ms, Queueing time: mean = 117.543 us, max = 296.469 us, min = 14.122 us, total = 170.203 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1448 total (0 active), Execution time: mean = 685.578 us, total = 992.717 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 726 total (1 active), Execution time: mean = 1.422 ms, total = 1.032 s, Queueing time: mean = 73.008 us, max = 323.875 us, min = -0.000 s, total = 53.004 ms GCSServer.deadline_timer.debug_state_event_stats_print - 121 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 193.984 ms, Queueing time: mean = 65.277 us, max = 182.019 us, min = 12.907 us, total = 7.899 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:02:54,006 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 335193 total (1 active) Queueing time: mean = 50.096 us, max = 7.565 ms, min = 1.647 us, total = 16.792 s Execution time: mean = 257.302 us, total = 86.246 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 152355 total (0 active), Execution time: mean = 531.959 us, total = 81.047 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 152355 total (0 active), Execution time: mean = 28.448 us, total = 4.334 s, Queueing time: mean = 107.436 us, max = 7.565 ms, min = 3.372 us, total = 16.368 s event_loop_lag_probe - 29029 total (0 active), Execution time: mean = 29.219 us, total = 848.184 ms, Queueing time: mean = 10.382 us, max = 4.983 ms, min = 1.647 us, total = 301.389 ms GcsTaskManager.GcJobSummary - 1453 total (1 active), Execution time: mean = 11.722 us, total = 17.032 ms, Queueing time: mean = 83.891 us, max = 338.131 us, min = 16.725 us, total = 121.894 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:02:54,007 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35693 total (25 active) Queueing time: mean = 19.704 us, max = 333.684 us, min = 2.330 us, total = 703.297 ms Execution time: mean = 4.799 s, total = 171278.948 s Event stats: event_loop_lag_probe - 29029 total (0 active), Execution time: mean = 31.913 us, total = 926.388 ms, Queueing time: mean = 11.758 us, max = 291.078 us, min = 2.330 us, total = 341.316 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1797 total (0 active), Execution time: mean = 43.895 us, total = 78.879 ms, Queueing time: mean = 105.690 us, max = 333.684 us, min = 6.593 us, total = 189.925 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1797 total (24 active), Execution time: mean = 95.313 s, total = 171276.757 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1497 total (0 active), Execution time: mean = 117.071 us, total = 175.255 ms, Queueing time: mean = 111.182 us, max = 293.835 us, min = 16.251 us, total = 166.439 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1497 total (0 active), Execution time: mean = 653.079 us, total = 977.659 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 651.471 us, total = 16.287 ms, Queueing time: mean = 87.988 us, max = 153.556 us, min = 48.361 us, total = 2.200 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:02:54,007 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29057 total (0 active) Queueing time: mean = 9.936 us, max = 1.423 ms, min = 568.000 ns, total = 288.699 ms Execution time: mean = 26.987 us, total = 784.152 ms Event stats: event_loop_lag_probe - 29031 total (0 active), Execution time: mean = 26.957 us, total = 782.603 ms, Queueing time: mean = 9.880 us, max = 1.423 ms, min = 1.850 us, total = 286.820 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:03:54,007 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1460 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1510 - cumulative published bytes: 1200821 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:03:54,008 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 88086 total (4 active) Queueing time: mean = 70.242 us, max = 279.303 ms, min = -0.000 s, total = 6.187 s Execution time: mean = 245.089 us, total = 21.589 s Event stats: event_loop_lag_probe - 29270 total (0 active), Execution time: mean = 25.344 us, total = 741.809 ms, Queueing time: mean = 10.044 us, max = 13.097 ms, min = 1.994 us, total = 293.996 ms GcsInMemoryStore.Put - 11750 total (0 active), Execution time: mean = 29.261 us, total = 343.817 ms, Queueing time: mean = 212.588 us, max = 278.545 ms, min = 1.649 us, total = 2.498 s RayletLoadPulled - 7318 total (1 active), Execution time: mean = 248.956 us, total = 1.822 s, Queueing time: mean = 75.597 us, max = 472.090 us, min = -0.000 s, total = 553.220 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7315 total (0 active), Execution time: mean = 77.337 us, total = 565.721 ms, Queueing time: mean = 103.590 us, max = 2.744 ms, min = 11.645 us, total = 757.758 ms NodeManagerService.grpc_client.GetResourceLoad - 7315 total (0 active), Execution time: mean = 1.494 ms, total = 10.931 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2934 total (0 active), Execution time: mean = 43.385 us, total = 127.291 ms, Queueing time: mean = 104.602 us, max = 479.211 us, min = 11.443 us, total = 306.902 ms InternalKVGcsService.grpc_server.InternalKVPut - 2934 total (0 active), Execution time: mean = 569.838 us, total = 1.672 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2927 total (0 active), Execution time: mean = 36.083 us, total = 105.616 ms, Queueing time: mean = 117.727 us, max = 1.498 ms, min = 20.329 us, total = 344.586 ms NodeInfoGcsService.grpc_server.CheckAlive - 2927 total (0 active), Execution time: mean = 595.677 us, total = 1.744 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2441 total (1 active), Execution time: mean = 9.605 us, total = 23.445 ms, Queueing time: mean = 75.936 us, max = 391.368 us, min = 10.687 us, total = 185.361 ms HealthCheck - 2437 total (0 active), Execution time: mean = 7.721 us, total = 18.816 ms, Queueing time: mean = 103.648 us, max = 564.605 us, min = 16.806 us, total = 252.590 ms GcsInMemoryStore.Get - 1529 total (0 active), Execution time: mean = 21.026 us, total = 32.149 ms, Queueing time: mean = 13.528 us, max = 128.508 us, min = 3.701 us, total = 20.684 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1528 total (0 active), Execution time: mean = 58.501 us, total = 89.389 ms, Queueing time: mean = 105.820 us, max = 269.544 us, min = 13.461 us, total = 161.694 ms InternalKVGcsService.grpc_server.InternalKVGet - 1528 total (0 active), Execution time: mean = 613.683 us, total = 937.707 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1460 total (0 active), Execution time: mean = 93.704 us, total = 136.808 ms, Queueing time: mean = 117.710 us, max = 296.469 us, min = 14.122 us, total = 171.856 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1460 total (0 active), Execution time: mean = 685.777 us, total = 1.001 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 732 total (1 active), Execution time: mean = 1.422 ms, total = 1.041 s, Queueing time: mean = 72.994 us, max = 323.875 us, min = -0.000 s, total = 53.432 ms GCSServer.deadline_timer.debug_state_event_stats_print - 122 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 195.705 ms, Queueing time: mean = 65.307 us, max = 182.019 us, min = 12.907 us, total = 7.967 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:03:54,008 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 337965 total (1 active) Queueing time: mean = 50.114 us, max = 7.565 ms, min = 1.647 us, total = 16.937 s Execution time: mean = 257.390 us, total = 86.989 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 153615 total (0 active), Execution time: mean = 532.131 us, total = 81.743 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 153615 total (0 active), Execution time: mean = 28.470 us, total = 4.373 s, Queueing time: mean = 107.476 us, max = 7.565 ms, min = 3.372 us, total = 16.510 s event_loop_lag_probe - 29269 total (0 active), Execution time: mean = 29.202 us, total = 854.700 ms, Queueing time: mean = 10.377 us, max = 4.983 ms, min = 1.647 us, total = 303.734 ms GcsTaskManager.GcJobSummary - 1465 total (1 active), Execution time: mean = 11.730 us, total = 17.184 ms, Queueing time: mean = 84.014 us, max = 338.131 us, min = 16.725 us, total = 123.080 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:03:54,008 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 35985 total (25 active) Queueing time: mean = 19.703 us, max = 333.684 us, min = 2.330 us, total = 709.005 ms Execution time: mean = 4.761 s, total = 171339.166 s Event stats: event_loop_lag_probe - 29269 total (0 active), Execution time: mean = 31.915 us, total = 934.124 ms, Queueing time: mean = 11.758 us, max = 291.078 us, min = 2.330 us, total = 344.154 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1810 total (0 active), Execution time: mean = 43.920 us, total = 79.495 ms, Queueing time: mean = 105.663 us, max = 333.684 us, min = 6.593 us, total = 191.251 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1810 total (24 active), Execution time: mean = 94.661 s, total = 171336.957 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1510 total (0 active), Execution time: mean = 116.875 us, total = 176.481 ms, Queueing time: mean = 111.248 us, max = 293.835 us, min = 16.251 us, total = 167.984 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1510 total (0 active), Execution time: mean = 652.704 us, total = 985.583 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 651.471 us, total = 16.287 ms, Queueing time: mean = 87.988 us, max = 153.556 us, min = 48.361 us, total = 2.200 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:03:54,008 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29297 total (0 active) Queueing time: mean = 9.945 us, max = 1.423 ms, min = 568.000 ns, total = 291.369 ms Execution time: mean = 27.008 us, total = 791.265 ms Event stats: event_loop_lag_probe - 29271 total (0 active), Execution time: mean = 26.979 us, total = 789.716 ms, Queueing time: mean = 9.890 us, max = 1.423 ms, min = 1.850 us, total = 289.489 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:04:54,009 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1472 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1522 - cumulative published bytes: 1210649 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:04:54,010 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 88805 total (4 active) Queueing time: mean = 69.986 us, max = 279.303 ms, min = -0.000 s, total = 6.215 s Execution time: mean = 245.071 us, total = 21.764 s Event stats: event_loop_lag_probe - 29510 total (0 active), Execution time: mean = 25.346 us, total = 747.970 ms, Queueing time: mean = 10.039 us, max = 13.097 ms, min = 1.994 us, total = 296.245 ms GcsInMemoryStore.Put - 11846 total (0 active), Execution time: mean = 29.065 us, total = 344.308 ms, Queueing time: mean = 211.225 us, max = 278.545 ms, min = 1.649 us, total = 2.502 s RayletLoadPulled - 7378 total (1 active), Execution time: mean = 248.997 us, total = 1.837 s, Queueing time: mean = 75.583 us, max = 472.090 us, min = -0.000 s, total = 557.650 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7375 total (0 active), Execution time: mean = 77.355 us, total = 570.496 ms, Queueing time: mean = 103.513 us, max = 2.744 ms, min = 11.645 us, total = 763.406 ms NodeManagerService.grpc_client.GetResourceLoad - 7375 total (0 active), Execution time: mean = 1.495 ms, total = 11.024 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2958 total (0 active), Execution time: mean = 43.366 us, total = 128.277 ms, Queueing time: mean = 104.544 us, max = 479.211 us, min = 11.443 us, total = 309.242 ms InternalKVGcsService.grpc_server.InternalKVPut - 2958 total (0 active), Execution time: mean = 569.768 us, total = 1.685 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2951 total (0 active), Execution time: mean = 36.038 us, total = 106.348 ms, Queueing time: mean = 117.652 us, max = 1.498 ms, min = 20.329 us, total = 347.191 ms NodeInfoGcsService.grpc_server.CheckAlive - 2951 total (0 active), Execution time: mean = 595.319 us, total = 1.757 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2461 total (1 active), Execution time: mean = 9.602 us, total = 23.630 ms, Queueing time: mean = 75.906 us, max = 391.368 us, min = 10.687 us, total = 186.805 ms HealthCheck - 2457 total (0 active), Execution time: mean = 7.715 us, total = 18.957 ms, Queueing time: mean = 103.471 us, max = 564.605 us, min = 16.806 us, total = 254.229 ms GcsInMemoryStore.Get - 1541 total (0 active), Execution time: mean = 21.002 us, total = 32.365 ms, Queueing time: mean = 13.540 us, max = 128.508 us, min = 3.701 us, total = 20.865 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1540 total (0 active), Execution time: mean = 58.622 us, total = 90.278 ms, Queueing time: mean = 105.698 us, max = 269.544 us, min = 13.461 us, total = 162.775 ms InternalKVGcsService.grpc_server.InternalKVGet - 1540 total (0 active), Execution time: mean = 613.625 us, total = 944.982 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1472 total (0 active), Execution time: mean = 93.649 us, total = 137.851 ms, Queueing time: mean = 117.705 us, max = 296.469 us, min = 14.122 us, total = 173.262 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1472 total (0 active), Execution time: mean = 685.567 us, total = 1.009 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 738 total (1 active), Execution time: mean = 1.421 ms, total = 1.049 s, Queueing time: mean = 72.929 us, max = 323.875 us, min = -0.000 s, total = 53.822 ms GCSServer.deadline_timer.debug_state_event_stats_print - 123 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 197.229 ms, Queueing time: mean = 65.184 us, max = 182.019 us, min = 12.907 us, total = 8.018 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:04:54,010 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 340737 total (1 active) Queueing time: mean = 50.124 us, max = 7.565 ms, min = 1.647 us, total = 17.079 s Execution time: mean = 257.475 us, total = 87.731 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 154875 total (0 active), Execution time: mean = 532.301 us, total = 82.440 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 154875 total (0 active), Execution time: mean = 28.489 us, total = 4.412 s, Queueing time: mean = 107.499 us, max = 7.565 ms, min = 3.372 us, total = 16.649 s event_loop_lag_probe - 29509 total (0 active), Execution time: mean = 29.196 us, total = 861.555 ms, Queueing time: mean = 10.372 us, max = 4.983 ms, min = 1.647 us, total = 306.060 ms GcsTaskManager.GcJobSummary - 1477 total (1 active), Execution time: mean = 11.732 us, total = 17.328 ms, Queueing time: mean = 84.024 us, max = 338.131 us, min = 16.725 us, total = 124.104 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:04:54,010 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36273 total (25 active) Queueing time: mean = 19.702 us, max = 333.684 us, min = 2.330 us, total = 714.645 ms Execution time: mean = 4.725 s, total = 171399.336 s Event stats: event_loop_lag_probe - 29509 total (0 active), Execution time: mean = 31.930 us, total = 942.236 ms, Queueing time: mean = 11.760 us, max = 291.078 us, min = 2.330 us, total = 347.033 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1822 total (0 active), Execution time: mean = 44.025 us, total = 80.214 ms, Queueing time: mean = 105.728 us, max = 333.684 us, min = 6.593 us, total = 192.636 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1822 total (24 active), Execution time: mean = 94.071 s, total = 171397.109 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1522 total (0 active), Execution time: mean = 116.943 us, total = 177.988 ms, Queueing time: mean = 111.274 us, max = 293.835 us, min = 16.251 us, total = 169.359 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1522 total (0 active), Execution time: mean = 652.991 us, total = 993.852 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 651.471 us, total = 16.287 ms, Queueing time: mean = 87.988 us, max = 153.556 us, min = 48.361 us, total = 2.200 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:04:54,010 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29537 total (0 active) Queueing time: mean = 9.939 us, max = 1.423 ms, min = 568.000 ns, total = 293.572 ms Execution time: mean = 26.994 us, total = 797.313 ms Event stats: event_loop_lag_probe - 29511 total (0 active), Execution time: mean = 26.965 us, total = 795.764 ms, Queueing time: mean = 9.884 us, max = 1.423 ms, min = 1.850 us, total = 291.693 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:05:54,010 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1484 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1534 - cumulative published bytes: 1220477 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:05:54,011 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 89524 total (4 active) Queueing time: mean = 69.790 us, max = 279.303 ms, min = -0.000 s, total = 6.248 s Execution time: mean = 245.043 us, total = 21.937 s Event stats: event_loop_lag_probe - 29750 total (0 active), Execution time: mean = 25.359 us, total = 754.435 ms, Queueing time: mean = 10.051 us, max = 13.097 ms, min = 1.994 us, total = 299.008 ms GcsInMemoryStore.Put - 11942 total (0 active), Execution time: mean = 28.873 us, total = 344.804 ms, Queueing time: mean = 209.922 us, max = 278.545 ms, min = 1.649 us, total = 2.507 s RayletLoadPulled - 7438 total (1 active), Execution time: mean = 248.920 us, total = 1.851 s, Queueing time: mean = 75.539 us, max = 472.090 us, min = -0.000 s, total = 561.856 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7435 total (0 active), Execution time: mean = 77.374 us, total = 575.275 ms, Queueing time: mean = 103.599 us, max = 2.744 ms, min = 11.645 us, total = 770.259 ms NodeManagerService.grpc_client.GetResourceLoad - 7435 total (0 active), Execution time: mean = 1.495 ms, total = 11.115 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 2982 total (0 active), Execution time: mean = 43.372 us, total = 129.335 ms, Queueing time: mean = 104.641 us, max = 479.211 us, min = 11.443 us, total = 312.040 ms InternalKVGcsService.grpc_server.InternalKVPut - 2982 total (0 active), Execution time: mean = 569.762 us, total = 1.699 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2975 total (0 active), Execution time: mean = 36.002 us, total = 107.107 ms, Queueing time: mean = 117.653 us, max = 1.498 ms, min = 20.329 us, total = 350.018 ms NodeInfoGcsService.grpc_server.CheckAlive - 2975 total (0 active), Execution time: mean = 594.954 us, total = 1.770 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2481 total (1 active), Execution time: mean = 9.594 us, total = 23.804 ms, Queueing time: mean = 75.920 us, max = 391.368 us, min = 10.687 us, total = 188.357 ms HealthCheck - 2477 total (0 active), Execution time: mean = 7.715 us, total = 19.110 ms, Queueing time: mean = 104.011 us, max = 1.431 ms, min = 16.806 us, total = 257.634 ms GcsInMemoryStore.Get - 1553 total (0 active), Execution time: mean = 20.996 us, total = 32.607 ms, Queueing time: mean = 13.540 us, max = 128.508 us, min = 3.701 us, total = 21.027 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1552 total (0 active), Execution time: mean = 58.598 us, total = 90.943 ms, Queueing time: mean = 105.815 us, max = 269.544 us, min = 13.461 us, total = 164.225 ms InternalKVGcsService.grpc_server.InternalKVGet - 1552 total (0 active), Execution time: mean = 613.516 us, total = 952.176 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1484 total (0 active), Execution time: mean = 93.548 us, total = 138.826 ms, Queueing time: mean = 117.811 us, max = 296.469 us, min = 14.122 us, total = 174.831 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1484 total (0 active), Execution time: mean = 685.543 us, total = 1.017 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 744 total (1 active), Execution time: mean = 1.421 ms, total = 1.057 s, Queueing time: mean = 72.902 us, max = 323.875 us, min = -0.000 s, total = 54.239 ms GCSServer.deadline_timer.debug_state_event_stats_print - 124 total (1 active, 1 running), Execution time: mean = 1.605 ms, total = 199.080 ms, Queueing time: mean = 65.205 us, max = 182.019 us, min = 12.907 us, total = 8.085 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:05:54,011 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 343508 total (1 active) Queueing time: mean = 50.129 us, max = 7.565 ms, min = 1.647 us, total = 17.220 s Execution time: mean = 257.493 us, total = 88.451 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 156135 total (0 active), Execution time: mean = 532.335 us, total = 83.116 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 156135 total (0 active), Execution time: mean = 28.494 us, total = 4.449 s, Queueing time: mean = 107.510 us, max = 7.565 ms, min = 3.372 us, total = 16.786 s event_loop_lag_probe - 29748 total (0 active), Execution time: mean = 29.190 us, total = 868.355 ms, Queueing time: mean = 10.367 us, max = 4.983 ms, min = 1.647 us, total = 308.396 ms GcsTaskManager.GcJobSummary - 1489 total (1 active), Execution time: mean = 11.722 us, total = 17.454 ms, Queueing time: mean = 83.939 us, max = 338.131 us, min = 16.725 us, total = 124.985 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:05:54,011 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36560 total (25 active) Queueing time: mean = 19.693 us, max = 333.684 us, min = 2.330 us, total = 719.969 ms Execution time: mean = 4.690 s, total = 171459.501 s Event stats: event_loop_lag_probe - 29748 total (0 active), Execution time: mean = 31.929 us, total = 949.816 ms, Queueing time: mean = 11.760 us, max = 291.078 us, min = 2.330 us, total = 349.822 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1834 total (0 active), Execution time: mean = 44.041 us, total = 80.771 ms, Queueing time: mean = 105.702 us, max = 333.684 us, min = 6.593 us, total = 193.858 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1834 total (24 active), Execution time: mean = 93.488 s, total = 171457.257 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1534 total (0 active), Execution time: mean = 116.884 us, total = 179.299 ms, Queueing time: mean = 111.259 us, max = 293.835 us, min = 16.251 us, total = 170.672 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1534 total (0 active), Execution time: mean = 652.871 us, total = 1.002 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 25 total (1 active), Execution time: mean = 651.471 us, total = 16.287 ms, Queueing time: mean = 87.988 us, max = 153.556 us, min = 48.361 us, total = 2.200 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:05:54,011 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 29777 total (0 active) Queueing time: mean = 9.939 us, max = 1.423 ms, min = 568.000 ns, total = 295.942 ms Execution time: mean = 26.988 us, total = 803.633 ms Event stats: event_loop_lag_probe - 29751 total (0 active), Execution time: mean = 26.960 us, total = 802.084 ms, Queueing time: mean = 9.884 us, max = 1.423 ms, min = 1.850 us, total = 294.063 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:06:54,011 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1496 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1546 - cumulative published bytes: 1230305 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:06:54,012 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 90243 total (4 active) Queueing time: mean = 69.534 us, max = 279.303 ms, min = -0.000 s, total = 6.275 s Execution time: mean = 244.966 us, total = 22.106 s Event stats: event_loop_lag_probe - 29990 total (0 active), Execution time: mean = 25.360 us, total = 760.533 ms, Queueing time: mean = 10.054 us, max = 13.097 ms, min = 1.994 us, total = 301.527 ms GcsInMemoryStore.Put - 12038 total (0 active), Execution time: mean = 28.684 us, total = 345.293 ms, Queueing time: mean = 208.619 us, max = 278.545 ms, min = 1.649 us, total = 2.511 s RayletLoadPulled - 7498 total (1 active), Execution time: mean = 248.854 us, total = 1.866 s, Queueing time: mean = 75.405 us, max = 472.090 us, min = -0.000 s, total = 565.384 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7495 total (0 active), Execution time: mean = 77.366 us, total = 579.858 ms, Queueing time: mean = 103.552 us, max = 2.744 ms, min = 11.645 us, total = 776.125 ms NodeManagerService.grpc_client.GetResourceLoad - 7495 total (0 active), Execution time: mean = 1.495 ms, total = 11.203 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3006 total (0 active), Execution time: mean = 43.384 us, total = 130.413 ms, Queueing time: mean = 104.556 us, max = 479.211 us, min = 11.443 us, total = 314.294 ms InternalKVGcsService.grpc_server.InternalKVPut - 3006 total (0 active), Execution time: mean = 569.503 us, total = 1.712 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 2999 total (0 active), Execution time: mean = 35.960 us, total = 107.845 ms, Queueing time: mean = 117.582 us, max = 1.498 ms, min = 20.329 us, total = 352.629 ms NodeInfoGcsService.grpc_server.CheckAlive - 2999 total (0 active), Execution time: mean = 594.618 us, total = 1.783 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2501 total (1 active), Execution time: mean = 9.587 us, total = 23.977 ms, Queueing time: mean = 75.841 us, max = 391.368 us, min = 10.687 us, total = 189.678 ms HealthCheck - 2497 total (0 active), Execution time: mean = 7.714 us, total = 19.262 ms, Queueing time: mean = 103.848 us, max = 1.431 ms, min = 16.806 us, total = 259.308 ms GcsInMemoryStore.Get - 1565 total (0 active), Execution time: mean = 21.013 us, total = 32.885 ms, Queueing time: mean = 13.529 us, max = 128.508 us, min = 3.701 us, total = 21.173 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1564 total (0 active), Execution time: mean = 58.561 us, total = 91.589 ms, Queueing time: mean = 105.672 us, max = 269.544 us, min = 13.461 us, total = 165.271 ms InternalKVGcsService.grpc_server.InternalKVGet - 1564 total (0 active), Execution time: mean = 613.268 us, total = 959.152 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1496 total (0 active), Execution time: mean = 93.573 us, total = 139.986 ms, Queueing time: mean = 117.706 us, max = 296.469 us, min = 14.122 us, total = 176.088 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1496 total (0 active), Execution time: mean = 685.475 us, total = 1.025 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 750 total (1 active), Execution time: mean = 1.421 ms, total = 1.066 s, Queueing time: mean = 72.818 us, max = 323.875 us, min = -0.000 s, total = 54.614 ms GCSServer.deadline_timer.debug_state_event_stats_print - 125 total (1 active, 1 running), Execution time: mean = 1.599 ms, total = 199.925 ms, Queueing time: mean = 65.172 us, max = 182.019 us, min = 12.907 us, total = 8.147 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:06:54,013 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 346280 total (1 active) Queueing time: mean = 50.119 us, max = 7.565 ms, min = 1.647 us, total = 17.355 s Execution time: mean = 257.525 us, total = 89.176 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 157395 total (0 active), Execution time: mean = 532.401 us, total = 83.797 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 157395 total (0 active), Execution time: mean = 28.499 us, total = 4.486 s, Queueing time: mean = 107.491 us, max = 7.565 ms, min = 3.372 us, total = 16.918 s event_loop_lag_probe - 29988 total (0 active), Execution time: mean = 29.185 us, total = 875.194 ms, Queueing time: mean = 10.363 us, max = 4.983 ms, min = 1.647 us, total = 310.763 ms GcsTaskManager.GcJobSummary - 1501 total (1 active), Execution time: mean = 11.721 us, total = 17.593 ms, Queueing time: mean = 83.890 us, max = 338.131 us, min = 16.725 us, total = 125.918 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:06:54,013 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 36859 total (25 active) Queueing time: mean = 19.688 us, max = 333.684 us, min = 2.330 us, total = 725.684 ms Execution time: mean = 4.735 s, total = 174519.679 s Event stats: event_loop_lag_probe - 29988 total (0 active), Execution time: mean = 31.913 us, total = 957.007 ms, Queueing time: mean = 11.752 us, max = 291.078 us, min = 2.330 us, total = 352.418 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1851 total (0 active), Execution time: mean = 44.020 us, total = 81.480 ms, Queueing time: mean = 105.664 us, max = 333.684 us, min = 6.593 us, total = 195.584 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1851 total (24 active), Execution time: mean = 94.283 s, total = 174517.417 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1546 total (0 active), Execution time: mean = 116.973 us, total = 180.841 ms, Queueing time: mean = 111.243 us, max = 293.835 us, min = 16.251 us, total = 171.981 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1546 total (0 active), Execution time: mean = 652.967 us, total = 1.009 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 633.525 us, total = 16.472 ms, Queueing time: mean = 87.854 us, max = 153.556 us, min = 48.361 us, total = 2.284 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:06:54,013 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30016 total (0 active) Queueing time: mean = 9.931 us, max = 1.423 ms, min = 568.000 ns, total = 298.075 ms Execution time: mean = 26.964 us, total = 809.350 ms Event stats: event_loop_lag_probe - 29990 total (0 active), Execution time: mean = 26.936 us, total = 807.801 ms, Queueing time: mean = 9.876 us, max = 1.423 ms, min = 1.850 us, total = 296.196 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:07:54,013 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1508 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1559 - cumulative published bytes: 1240167 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:07:54,014 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 90962 total (4 active) Queueing time: mean = 69.336 us, max = 279.303 ms, min = -0.000 s, total = 6.307 s Execution time: mean = 245.080 us, total = 22.293 s Event stats: event_loop_lag_probe - 30230 total (0 active), Execution time: mean = 25.382 us, total = 767.288 ms, Queueing time: mean = 10.055 us, max = 13.097 ms, min = 1.994 us, total = 303.955 ms GcsInMemoryStore.Put - 12134 total (0 active), Execution time: mean = 28.501 us, total = 345.835 ms, Queueing time: mean = 207.346 us, max = 278.545 ms, min = 1.649 us, total = 2.516 s RayletLoadPulled - 7558 total (1 active), Execution time: mean = 249.027 us, total = 1.882 s, Queueing time: mean = 75.518 us, max = 472.090 us, min = -0.000 s, total = 570.761 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7555 total (0 active), Execution time: mean = 77.437 us, total = 585.036 ms, Queueing time: mean = 103.596 us, max = 2.744 ms, min = 11.645 us, total = 782.665 ms NodeManagerService.grpc_client.GetResourceLoad - 7555 total (0 active), Execution time: mean = 1.495 ms, total = 11.297 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3030 total (0 active), Execution time: mean = 43.367 us, total = 131.402 ms, Queueing time: mean = 104.644 us, max = 479.211 us, min = 11.443 us, total = 317.071 ms InternalKVGcsService.grpc_server.InternalKVPut - 3030 total (0 active), Execution time: mean = 569.660 us, total = 1.726 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3023 total (0 active), Execution time: mean = 35.971 us, total = 108.739 ms, Queueing time: mean = 117.668 us, max = 1.498 ms, min = 20.329 us, total = 355.711 ms NodeInfoGcsService.grpc_server.CheckAlive - 3023 total (0 active), Execution time: mean = 594.652 us, total = 1.798 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2521 total (1 active), Execution time: mean = 9.588 us, total = 24.173 ms, Queueing time: mean = 75.901 us, max = 391.368 us, min = 10.687 us, total = 191.346 ms HealthCheck - 2517 total (0 active), Execution time: mean = 7.721 us, total = 19.434 ms, Queueing time: mean = 103.898 us, max = 1.431 ms, min = 16.806 us, total = 261.512 ms GcsInMemoryStore.Get - 1577 total (0 active), Execution time: mean = 21.006 us, total = 33.126 ms, Queueing time: mean = 13.527 us, max = 128.508 us, min = 3.701 us, total = 21.332 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1576 total (0 active), Execution time: mean = 58.573 us, total = 92.311 ms, Queueing time: mean = 105.634 us, max = 269.544 us, min = 13.461 us, total = 166.479 ms InternalKVGcsService.grpc_server.InternalKVGet - 1576 total (0 active), Execution time: mean = 616.188 us, total = 971.112 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1508 total (0 active), Execution time: mean = 93.515 us, total = 141.021 ms, Queueing time: mean = 117.705 us, max = 296.469 us, min = 14.122 us, total = 177.499 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1508 total (0 active), Execution time: mean = 685.554 us, total = 1.034 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 756 total (1 active), Execution time: mean = 1.422 ms, total = 1.075 s, Queueing time: mean = 72.867 us, max = 323.875 us, min = -0.000 s, total = 55.088 ms GCSServer.deadline_timer.debug_state_event_stats_print - 126 total (1 active, 1 running), Execution time: mean = 1.600 ms, total = 201.628 ms, Queueing time: mean = 65.192 us, max = 182.019 us, min = 12.907 us, total = 8.214 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:07:54,014 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 349052 total (1 active) Queueing time: mean = 50.127 us, max = 7.565 ms, min = 1.647 us, total = 17.497 s Execution time: mean = 257.610 us, total = 89.919 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 158655 total (0 active), Execution time: mean = 532.560 us, total = 84.493 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 158655 total (0 active), Execution time: mean = 28.525 us, total = 4.526 s, Queueing time: mean = 107.508 us, max = 7.565 ms, min = 3.372 us, total = 17.057 s event_loop_lag_probe - 30228 total (0 active), Execution time: mean = 29.194 us, total = 882.483 ms, Queueing time: mean = 10.366 us, max = 4.983 ms, min = 1.647 us, total = 313.343 ms GcsTaskManager.GcJobSummary - 1513 total (1 active), Execution time: mean = 11.728 us, total = 17.744 ms, Queueing time: mean = 83.914 us, max = 338.131 us, min = 16.725 us, total = 126.961 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:07:54,015 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37151 total (25 active) Queueing time: mean = 19.686 us, max = 333.684 us, min = 2.330 us, total = 731.352 ms Execution time: mean = 4.699 s, total = 174579.900 s Event stats: event_loop_lag_probe - 30228 total (0 active), Execution time: mean = 31.913 us, total = 964.655 ms, Queueing time: mean = 11.752 us, max = 291.078 us, min = 2.330 us, total = 355.234 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1864 total (0 active), Execution time: mean = 44.073 us, total = 82.152 ms, Queueing time: mean = 105.581 us, max = 333.684 us, min = 6.593 us, total = 196.802 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1864 total (24 active), Execution time: mean = 93.658 s, total = 174577.619 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1559 total (0 active), Execution time: mean = 116.913 us, total = 182.267 ms, Queueing time: mean = 111.362 us, max = 293.835 us, min = 16.251 us, total = 173.613 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1559 total (0 active), Execution time: mean = 653.049 us, total = 1.018 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 633.525 us, total = 16.472 ms, Queueing time: mean = 87.854 us, max = 153.556 us, min = 48.361 us, total = 2.284 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:07:54,015 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30256 total (0 active) Queueing time: mean = 9.933 us, max = 1.423 ms, min = 568.000 ns, total = 300.546 ms Execution time: mean = 26.969 us, total = 815.981 ms Event stats: event_loop_lag_probe - 30230 total (0 active), Execution time: mean = 26.941 us, total = 814.432 ms, Queueing time: mean = 9.880 us, max = 1.423 ms, min = 1.850 us, total = 298.666 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:08:54,015 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1520 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1571 - cumulative published bytes: 1249995 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:08:54,016 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 91681 total (4 active) Queueing time: mean = 69.130 us, max = 279.303 ms, min = -0.000 s, total = 6.338 s Execution time: mean = 245.114 us, total = 22.472 s Event stats: event_loop_lag_probe - 30470 total (0 active), Execution time: mean = 25.399 us, total = 773.893 ms, Queueing time: mean = 10.049 us, max = 13.097 ms, min = 1.994 us, total = 306.182 ms GcsInMemoryStore.Put - 12230 total (0 active), Execution time: mean = 28.321 us, total = 346.360 ms, Queueing time: mean = 206.103 us, max = 278.545 ms, min = 1.649 us, total = 2.521 s RayletLoadPulled - 7618 total (1 active), Execution time: mean = 249.033 us, total = 1.897 s, Queueing time: mean = 75.595 us, max = 472.090 us, min = -0.000 s, total = 575.883 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7615 total (0 active), Execution time: mean = 77.453 us, total = 589.804 ms, Queueing time: mean = 103.639 us, max = 2.744 ms, min = 11.645 us, total = 789.209 ms NodeManagerService.grpc_client.GetResourceLoad - 7615 total (0 active), Execution time: mean = 1.496 ms, total = 11.389 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3054 total (0 active), Execution time: mean = 43.568 us, total = 133.056 ms, Queueing time: mean = 104.648 us, max = 479.211 us, min = 11.443 us, total = 319.594 ms InternalKVGcsService.grpc_server.InternalKVPut - 3054 total (0 active), Execution time: mean = 569.882 us, total = 1.740 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3047 total (0 active), Execution time: mean = 36.017 us, total = 109.745 ms, Queueing time: mean = 117.691 us, max = 1.498 ms, min = 20.329 us, total = 358.604 ms NodeInfoGcsService.grpc_server.CheckAlive - 3047 total (0 active), Execution time: mean = 594.748 us, total = 1.812 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2541 total (1 active), Execution time: mean = 9.588 us, total = 24.364 ms, Queueing time: mean = 75.933 us, max = 391.368 us, min = 10.687 us, total = 192.946 ms HealthCheck - 2537 total (0 active), Execution time: mean = 7.717 us, total = 19.578 ms, Queueing time: mean = 103.775 us, max = 1.431 ms, min = 16.806 us, total = 263.278 ms GcsInMemoryStore.Get - 1589 total (0 active), Execution time: mean = 21.024 us, total = 33.407 ms, Queueing time: mean = 13.531 us, max = 128.508 us, min = 3.701 us, total = 21.501 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1588 total (0 active), Execution time: mean = 58.758 us, total = 93.307 ms, Queueing time: mean = 105.674 us, max = 269.544 us, min = 13.461 us, total = 167.810 ms InternalKVGcsService.grpc_server.InternalKVGet - 1588 total (0 active), Execution time: mean = 616.383 us, total = 978.816 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1520 total (0 active), Execution time: mean = 93.482 us, total = 142.092 ms, Queueing time: mean = 117.751 us, max = 296.469 us, min = 14.122 us, total = 178.982 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1520 total (0 active), Execution time: mean = 685.390 us, total = 1.042 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 762 total (1 active), Execution time: mean = 1.422 ms, total = 1.084 s, Queueing time: mean = 72.950 us, max = 323.875 us, min = -0.000 s, total = 55.588 ms GCSServer.deadline_timer.debug_state_event_stats_print - 127 total (1 active, 1 running), Execution time: mean = 1.602 ms, total = 203.458 ms, Queueing time: mean = 65.195 us, max = 182.019 us, min = 12.907 us, total = 8.280 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:08:54,016 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 351824 total (1 active) Queueing time: mean = 50.132 us, max = 7.565 ms, min = 1.647 us, total = 17.637 s Execution time: mean = 257.695 us, total = 90.663 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 159915 total (0 active), Execution time: mean = 532.727 us, total = 85.191 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 159915 total (0 active), Execution time: mean = 28.545 us, total = 4.565 s, Queueing time: mean = 107.518 us, max = 7.565 ms, min = 3.372 us, total = 17.194 s event_loop_lag_probe - 30468 total (0 active), Execution time: mean = 29.192 us, total = 889.418 ms, Queueing time: mean = 10.363 us, max = 4.983 ms, min = 1.647 us, total = 315.749 ms GcsTaskManager.GcJobSummary - 1525 total (1 active), Execution time: mean = 11.732 us, total = 17.891 ms, Queueing time: mean = 83.901 us, max = 338.131 us, min = 16.725 us, total = 127.949 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:08:54,017 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37439 total (25 active) Queueing time: mean = 19.687 us, max = 333.684 us, min = 2.330 us, total = 737.071 ms Execution time: mean = 4.665 s, total = 174640.089 s Event stats: event_loop_lag_probe - 30468 total (0 active), Execution time: mean = 31.916 us, total = 972.412 ms, Queueing time: mean = 11.756 us, max = 291.078 us, min = 2.330 us, total = 358.179 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1876 total (0 active), Execution time: mean = 44.131 us, total = 82.790 ms, Queueing time: mean = 105.642 us, max = 333.684 us, min = 6.593 us, total = 198.185 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1876 total (24 active), Execution time: mean = 93.091 s, total = 174637.790 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1571 total (0 active), Execution time: mean = 116.896 us, total = 183.644 ms, Queueing time: mean = 111.398 us, max = 293.835 us, min = 16.251 us, total = 175.006 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1571 total (0 active), Execution time: mean = 653.040 us, total = 1.026 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 633.525 us, total = 16.472 ms, Queueing time: mean = 87.854 us, max = 153.556 us, min = 48.361 us, total = 2.284 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:08:54,017 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30496 total (0 active) Queueing time: mean = 9.929 us, max = 1.423 ms, min = 568.000 ns, total = 302.781 ms Execution time: mean = 26.957 us, total = 822.080 ms Event stats: event_loop_lag_probe - 30470 total (0 active), Execution time: mean = 26.929 us, total = 820.531 ms, Queueing time: mean = 9.875 us, max = 1.423 ms, min = 1.850 us, total = 300.902 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:09:54,017 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1532 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1583 - cumulative published bytes: 1259823 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:09:54,018 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 92399 total (4 active) Queueing time: mean = 68.928 us, max = 279.303 ms, min = -0.000 s, total = 6.369 s Execution time: mean = 245.146 us, total = 22.651 s Event stats: event_loop_lag_probe - 30709 total (0 active), Execution time: mean = 25.413 us, total = 780.402 ms, Queueing time: mean = 10.048 us, max = 13.097 ms, min = 1.994 us, total = 308.568 ms GcsInMemoryStore.Put - 12326 total (0 active), Execution time: mean = 28.142 us, total = 346.882 ms, Queueing time: mean = 204.890 us, max = 278.545 ms, min = 1.649 us, total = 2.525 s RayletLoadPulled - 7678 total (1 active), Execution time: mean = 249.074 us, total = 1.912 s, Queueing time: mean = 75.636 us, max = 472.090 us, min = -0.000 s, total = 580.734 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7675 total (0 active), Execution time: mean = 77.460 us, total = 594.502 ms, Queueing time: mean = 103.633 us, max = 2.744 ms, min = 11.645 us, total = 795.380 ms NodeManagerService.grpc_client.GetResourceLoad - 7675 total (0 active), Execution time: mean = 1.496 ms, total = 11.482 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3078 total (0 active), Execution time: mean = 43.575 us, total = 134.124 ms, Queueing time: mean = 104.665 us, max = 479.211 us, min = 11.443 us, total = 322.159 ms InternalKVGcsService.grpc_server.InternalKVPut - 3078 total (0 active), Execution time: mean = 570.153 us, total = 1.755 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3071 total (0 active), Execution time: mean = 35.991 us, total = 110.530 ms, Queueing time: mean = 117.774 us, max = 1.498 ms, min = 20.329 us, total = 361.685 ms NodeInfoGcsService.grpc_server.CheckAlive - 3071 total (0 active), Execution time: mean = 594.794 us, total = 1.827 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2561 total (1 active), Execution time: mean = 9.599 us, total = 24.582 ms, Queueing time: mean = 75.957 us, max = 391.368 us, min = 10.687 us, total = 194.526 ms HealthCheck - 2557 total (0 active), Execution time: mean = 7.735 us, total = 19.778 ms, Queueing time: mean = 103.833 us, max = 1.431 ms, min = 16.806 us, total = 265.501 ms GcsInMemoryStore.Get - 1601 total (0 active), Execution time: mean = 21.010 us, total = 33.637 ms, Queueing time: mean = 13.526 us, max = 128.508 us, min = 3.701 us, total = 21.655 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1600 total (0 active), Execution time: mean = 58.782 us, total = 94.051 ms, Queueing time: mean = 105.701 us, max = 269.544 us, min = 13.461 us, total = 169.121 ms InternalKVGcsService.grpc_server.InternalKVGet - 1600 total (0 active), Execution time: mean = 616.338 us, total = 986.141 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1532 total (0 active), Execution time: mean = 93.387 us, total = 143.069 ms, Queueing time: mean = 117.753 us, max = 296.469 us, min = 14.122 us, total = 180.397 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1532 total (0 active), Execution time: mean = 685.089 us, total = 1.050 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 768 total (1 active), Execution time: mean = 1.423 ms, total = 1.093 s, Queueing time: mean = 72.835 us, max = 323.875 us, min = -0.000 s, total = 55.938 ms GCSServer.deadline_timer.debug_state_event_stats_print - 128 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 205.188 ms, Queueing time: mean = 65.105 us, max = 182.019 us, min = 12.907 us, total = 8.333 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:09:54,018 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 354596 total (1 active) Queueing time: mean = 50.146 us, max = 7.565 ms, min = 1.647 us, total = 17.782 s Execution time: mean = 257.773 us, total = 91.405 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 161175 total (0 active), Execution time: mean = 532.888 us, total = 85.888 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 161175 total (0 active), Execution time: mean = 28.557 us, total = 4.603 s, Queueing time: mean = 107.550 us, max = 7.565 ms, min = 3.372 us, total = 17.334 s event_loop_lag_probe - 30708 total (0 active), Execution time: mean = 29.193 us, total = 896.458 ms, Queueing time: mean = 10.365 us, max = 4.983 ms, min = 1.647 us, total = 318.293 ms GcsTaskManager.GcJobSummary - 1537 total (1 active), Execution time: mean = 11.716 us, total = 18.008 ms, Queueing time: mean = 83.832 us, max = 338.131 us, min = 16.725 us, total = 128.850 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:09:54,018 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 37727 total (25 active) Queueing time: mean = 19.688 us, max = 333.684 us, min = 2.330 us, total = 742.772 ms Execution time: mean = 4.631 s, total = 174700.199 s Event stats: event_loop_lag_probe - 30708 total (0 active), Execution time: mean = 31.927 us, total = 980.403 ms, Queueing time: mean = 11.761 us, max = 291.078 us, min = 2.330 us, total = 361.158 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1888 total (0 active), Execution time: mean = 44.131 us, total = 83.319 ms, Queueing time: mean = 105.681 us, max = 333.684 us, min = 6.593 us, total = 199.525 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1888 total (24 active), Execution time: mean = 92.531 s, total = 174697.883 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1583 total (0 active), Execution time: mean = 116.925 us, total = 185.092 ms, Queueing time: mean = 111.426 us, max = 293.835 us, min = 16.251 us, total = 176.387 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1583 total (0 active), Execution time: mean = 653.207 us, total = 1.034 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 633.525 us, total = 16.472 ms, Queueing time: mean = 87.854 us, max = 153.556 us, min = 48.361 us, total = 2.284 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:09:54,018 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30736 total (0 active) Queueing time: mean = 9.927 us, max = 1.423 ms, min = 568.000 ns, total = 305.102 ms Execution time: mean = 26.952 us, total = 828.386 ms Event stats: event_loop_lag_probe - 30710 total (0 active), Execution time: mean = 26.924 us, total = 826.837 ms, Queueing time: mean = 9.874 us, max = 1.423 ms, min = 1.850 us, total = 303.223 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:10:54,019 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1544 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1595 - cumulative published bytes: 1269651 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:10:54,020 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 93118 total (4 active) Queueing time: mean = 68.753 us, max = 279.303 ms, min = -0.000 s, total = 6.402 s Execution time: mean = 245.213 us, total = 22.834 s Event stats: event_loop_lag_probe - 30949 total (0 active), Execution time: mean = 25.432 us, total = 787.080 ms, Queueing time: mean = 10.048 us, max = 13.097 ms, min = 1.994 us, total = 310.969 ms GcsInMemoryStore.Put - 12422 total (0 active), Execution time: mean = 27.969 us, total = 347.431 ms, Queueing time: mean = 203.716 us, max = 278.545 ms, min = 1.649 us, total = 2.531 s RayletLoadPulled - 7738 total (1 active), Execution time: mean = 249.214 us, total = 1.928 s, Queueing time: mean = 75.706 us, max = 472.090 us, min = -0.000 s, total = 585.814 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7735 total (0 active), Execution time: mean = 77.490 us, total = 599.385 ms, Queueing time: mean = 103.683 us, max = 2.744 ms, min = 11.645 us, total = 801.989 ms NodeManagerService.grpc_client.GetResourceLoad - 7735 total (0 active), Execution time: mean = 1.497 ms, total = 11.577 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3102 total (0 active), Execution time: mean = 43.607 us, total = 135.269 ms, Queueing time: mean = 104.774 us, max = 479.211 us, min = 11.443 us, total = 325.010 ms InternalKVGcsService.grpc_server.InternalKVPut - 3102 total (0 active), Execution time: mean = 570.182 us, total = 1.769 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3095 total (0 active), Execution time: mean = 36.001 us, total = 111.423 ms, Queueing time: mean = 117.891 us, max = 1.498 ms, min = 20.329 us, total = 364.872 ms NodeInfoGcsService.grpc_server.CheckAlive - 3095 total (0 active), Execution time: mean = 594.882 us, total = 1.841 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2581 total (1 active), Execution time: mean = 9.609 us, total = 24.800 ms, Queueing time: mean = 76.068 us, max = 391.368 us, min = 10.687 us, total = 196.333 ms HealthCheck - 2577 total (0 active), Execution time: mean = 7.747 us, total = 19.965 ms, Queueing time: mean = 103.949 us, max = 1.431 ms, min = 16.806 us, total = 267.877 ms GcsInMemoryStore.Get - 1613 total (0 active), Execution time: mean = 21.022 us, total = 33.908 ms, Queueing time: mean = 13.529 us, max = 128.508 us, min = 3.701 us, total = 21.822 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1612 total (0 active), Execution time: mean = 58.801 us, total = 94.787 ms, Queueing time: mean = 105.817 us, max = 269.544 us, min = 13.461 us, total = 170.577 ms InternalKVGcsService.grpc_server.InternalKVGet - 1612 total (0 active), Execution time: mean = 616.519 us, total = 993.829 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1544 total (0 active), Execution time: mean = 93.414 us, total = 144.231 ms, Queueing time: mean = 117.929 us, max = 296.469 us, min = 14.122 us, total = 182.083 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1544 total (0 active), Execution time: mean = 685.234 us, total = 1.058 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 774 total (1 active), Execution time: mean = 1.423 ms, total = 1.102 s, Queueing time: mean = 72.893 us, max = 323.875 us, min = -0.000 s, total = 56.419 ms GCSServer.deadline_timer.debug_state_event_stats_print - 129 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 206.801 ms, Queueing time: mean = 65.081 us, max = 182.019 us, min = 12.907 us, total = 8.395 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:10:54,020 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 357368 total (1 active) Queueing time: mean = 50.162 us, max = 7.565 ms, min = 1.647 us, total = 17.926 s Execution time: mean = 257.869 us, total = 92.154 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 162435 total (0 active), Execution time: mean = 533.079 us, total = 86.591 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 162435 total (0 active), Execution time: mean = 28.577 us, total = 4.642 s, Queueing time: mean = 107.585 us, max = 7.565 ms, min = 3.372 us, total = 17.476 s event_loop_lag_probe - 30948 total (0 active), Execution time: mean = 29.194 us, total = 903.497 ms, Queueing time: mean = 10.364 us, max = 4.983 ms, min = 1.647 us, total = 320.738 ms GcsTaskManager.GcJobSummary - 1549 total (1 active), Execution time: mean = 11.725 us, total = 18.161 ms, Queueing time: mean = 83.944 us, max = 338.131 us, min = 16.725 us, total = 130.030 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:10:54,020 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38015 total (25 active) Queueing time: mean = 19.686 us, max = 333.684 us, min = 2.330 us, total = 748.369 ms Execution time: mean = 4.597 s, total = 174760.436 s Event stats: event_loop_lag_probe - 30948 total (0 active), Execution time: mean = 31.942 us, total = 988.534 ms, Queueing time: mean = 11.769 us, max = 291.078 us, min = 2.330 us, total = 364.227 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1900 total (0 active), Execution time: mean = 44.131 us, total = 83.849 ms, Queueing time: mean = 105.643 us, max = 333.684 us, min = 6.593 us, total = 200.721 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1900 total (24 active), Execution time: mean = 91.978 s, total = 174758.101 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1595 total (0 active), Execution time: mean = 117.045 us, total = 186.687 ms, Queueing time: mean = 111.423 us, max = 293.835 us, min = 16.251 us, total = 177.719 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1595 total (0 active), Execution time: mean = 653.428 us, total = 1.042 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 26 total (1 active), Execution time: mean = 633.525 us, total = 16.472 ms, Queueing time: mean = 87.854 us, max = 153.556 us, min = 48.361 us, total = 2.284 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:10:54,020 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 30976 total (0 active) Queueing time: mean = 9.925 us, max = 1.423 ms, min = 568.000 ns, total = 307.423 ms Execution time: mean = 26.951 us, total = 834.821 ms Event stats: event_loop_lag_probe - 30950 total (0 active), Execution time: mean = 26.923 us, total = 833.272 ms, Queueing time: mean = 9.872 us, max = 1.423 ms, min = 1.850 us, total = 305.544 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:11:54,020 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1556 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1607 - cumulative published bytes: 1279479 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:11:54,021 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 93837 total (4 active) Queueing time: mean = 68.533 us, max = 279.303 ms, min = -0.000 s, total = 6.431 s Execution time: mean = 245.214 us, total = 23.010 s Event stats: event_loop_lag_probe - 31189 total (0 active), Execution time: mean = 25.435 us, total = 793.279 ms, Queueing time: mean = 10.042 us, max = 13.097 ms, min = 1.994 us, total = 313.190 ms GcsInMemoryStore.Put - 12518 total (0 active), Execution time: mean = 27.798 us, total = 347.975 ms, Queueing time: mean = 202.486 us, max = 278.545 ms, min = 1.649 us, total = 2.535 s RayletLoadPulled - 7798 total (1 active), Execution time: mean = 249.137 us, total = 1.943 s, Queueing time: mean = 75.681 us, max = 472.090 us, min = -0.000 s, total = 590.158 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7795 total (0 active), Execution time: mean = 77.464 us, total = 603.831 ms, Queueing time: mean = 103.585 us, max = 2.744 ms, min = 11.645 us, total = 807.445 ms NodeManagerService.grpc_client.GetResourceLoad - 7795 total (0 active), Execution time: mean = 1.497 ms, total = 11.670 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3126 total (0 active), Execution time: mean = 43.631 us, total = 136.390 ms, Queueing time: mean = 104.863 us, max = 479.211 us, min = 11.443 us, total = 327.802 ms InternalKVGcsService.grpc_server.InternalKVPut - 3126 total (0 active), Execution time: mean = 570.320 us, total = 1.783 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3119 total (0 active), Execution time: mean = 36.004 us, total = 112.295 ms, Queueing time: mean = 117.876 us, max = 1.498 ms, min = 20.329 us, total = 367.656 ms NodeInfoGcsService.grpc_server.CheckAlive - 3119 total (0 active), Execution time: mean = 594.624 us, total = 1.855 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2601 total (1 active), Execution time: mean = 9.610 us, total = 24.995 ms, Queueing time: mean = 76.114 us, max = 391.368 us, min = 10.687 us, total = 197.973 ms HealthCheck - 2597 total (0 active), Execution time: mean = 7.748 us, total = 20.121 ms, Queueing time: mean = 103.885 us, max = 1.431 ms, min = 16.806 us, total = 269.788 ms GcsInMemoryStore.Get - 1625 total (0 active), Execution time: mean = 21.015 us, total = 34.149 ms, Queueing time: mean = 13.523 us, max = 128.508 us, min = 3.701 us, total = 21.975 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1624 total (0 active), Execution time: mean = 58.778 us, total = 95.456 ms, Queueing time: mean = 105.882 us, max = 269.544 us, min = 13.461 us, total = 171.952 ms InternalKVGcsService.grpc_server.InternalKVGet - 1624 total (0 active), Execution time: mean = 616.563 us, total = 1.001 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1556 total (0 active), Execution time: mean = 93.379 us, total = 145.297 ms, Queueing time: mean = 118.011 us, max = 296.469 us, min = 14.122 us, total = 183.625 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1556 total (0 active), Execution time: mean = 685.280 us, total = 1.066 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 780 total (1 active), Execution time: mean = 1.423 ms, total = 1.110 s, Queueing time: mean = 72.873 us, max = 323.875 us, min = -0.000 s, total = 56.841 ms GCSServer.deadline_timer.debug_state_event_stats_print - 130 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 208.504 ms, Queueing time: mean = 65.077 us, max = 182.019 us, min = 12.907 us, total = 8.460 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:11:54,021 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 360140 total (1 active) Queueing time: mean = 50.167 us, max = 7.565 ms, min = 1.647 us, total = 18.067 s Execution time: mean = 257.926 us, total = 92.890 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 163695 total (0 active), Execution time: mean = 533.185 us, total = 87.280 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 163695 total (0 active), Execution time: mean = 28.596 us, total = 4.681 s, Queueing time: mean = 107.594 us, max = 7.565 ms, min = 3.372 us, total = 17.613 s event_loop_lag_probe - 31188 total (0 active), Execution time: mean = 29.191 us, total = 910.398 ms, Queueing time: mean = 10.361 us, max = 4.983 ms, min = 1.647 us, total = 323.152 ms GcsTaskManager.GcJobSummary - 1561 total (1 active), Execution time: mean = 11.728 us, total = 18.307 ms, Queueing time: mean = 84.024 us, max = 338.131 us, min = 16.725 us, total = 131.162 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:11:54,021 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38340 total (25 active) Queueing time: mean = 19.705 us, max = 333.684 us, min = 2.330 us, total = 755.498 ms Execution time: mean = 4.841 s, total = 185620.529 s Event stats: event_loop_lag_probe - 31188 total (0 active), Execution time: mean = 31.956 us, total = 996.629 ms, Queueing time: mean = 11.775 us, max = 291.078 us, min = 2.330 us, total = 367.238 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1930 total (0 active), Execution time: mean = 43.960 us, total = 84.843 ms, Queueing time: mean = 105.420 us, max = 333.684 us, min = 6.593 us, total = 203.460 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1930 total (24 active), Execution time: mean = 96.175 s, total = 185618.174 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1607 total (0 active), Execution time: mean = 117.250 us, total = 188.421 ms, Queueing time: mean = 111.392 us, max = 293.835 us, min = 16.251 us, total = 179.007 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1607 total (0 active), Execution time: mean = 653.750 us, total = 1.051 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 632.327 us, total = 17.073 ms, Queueing time: mean = 87.982 us, max = 153.556 us, min = 48.361 us, total = 2.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:11:54,022 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31216 total (0 active) Queueing time: mean = 9.916 us, max = 1.423 ms, min = 568.000 ns, total = 309.546 ms Execution time: mean = 26.931 us, total = 840.692 ms Event stats: event_loop_lag_probe - 31190 total (0 active), Execution time: mean = 26.904 us, total = 839.143 ms, Queueing time: mean = 9.864 us, max = 1.423 ms, min = 1.850 us, total = 307.667 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:12:54,022 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1568 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1619 - cumulative published bytes: 1289307 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:12:54,023 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 94556 total (4 active) Queueing time: mean = 68.345 us, max = 279.303 ms, min = -0.000 s, total = 6.462 s Execution time: mean = 245.297 us, total = 23.194 s Event stats: event_loop_lag_probe - 31429 total (0 active), Execution time: mean = 25.451 us, total = 799.885 ms, Queueing time: mean = 10.039 us, max = 13.097 ms, min = 1.994 us, total = 315.526 ms GcsInMemoryStore.Put - 12614 total (0 active), Execution time: mean = 27.639 us, total = 348.633 ms, Queueing time: mean = 201.290 us, max = 278.545 ms, min = 1.649 us, total = 2.539 s RayletLoadPulled - 7858 total (1 active), Execution time: mean = 249.193 us, total = 1.958 s, Queueing time: mean = 75.740 us, max = 472.090 us, min = -0.000 s, total = 595.166 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7855 total (0 active), Execution time: mean = 77.487 us, total = 608.659 ms, Queueing time: mean = 103.633 us, max = 2.744 ms, min = 11.645 us, total = 814.037 ms NodeManagerService.grpc_client.GetResourceLoad - 7855 total (0 active), Execution time: mean = 1.498 ms, total = 11.768 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3150 total (0 active), Execution time: mean = 43.580 us, total = 137.276 ms, Queueing time: mean = 104.968 us, max = 479.211 us, min = 11.443 us, total = 330.650 ms InternalKVGcsService.grpc_server.InternalKVPut - 3150 total (0 active), Execution time: mean = 570.443 us, total = 1.797 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3143 total (0 active), Execution time: mean = 35.985 us, total = 113.102 ms, Queueing time: mean = 117.895 us, max = 1.498 ms, min = 20.329 us, total = 370.545 ms NodeInfoGcsService.grpc_server.CheckAlive - 3143 total (0 active), Execution time: mean = 594.673 us, total = 1.869 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2621 total (1 active), Execution time: mean = 9.611 us, total = 25.192 ms, Queueing time: mean = 76.145 us, max = 391.368 us, min = 10.687 us, total = 199.576 ms HealthCheck - 2617 total (0 active), Execution time: mean = 7.757 us, total = 20.301 ms, Queueing time: mean = 104.006 us, max = 1.431 ms, min = 16.806 us, total = 272.184 ms GcsInMemoryStore.Get - 1637 total (0 active), Execution time: mean = 21.006 us, total = 34.386 ms, Queueing time: mean = 13.533 us, max = 128.508 us, min = 3.701 us, total = 22.153 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1636 total (0 active), Execution time: mean = 58.794 us, total = 96.186 ms, Queueing time: mean = 105.842 us, max = 269.544 us, min = 13.461 us, total = 173.157 ms InternalKVGcsService.grpc_server.InternalKVGet - 1636 total (0 active), Execution time: mean = 616.509 us, total = 1.009 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1568 total (0 active), Execution time: mean = 93.325 us, total = 146.334 ms, Queueing time: mean = 118.058 us, max = 296.469 us, min = 14.122 us, total = 185.115 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1568 total (0 active), Execution time: mean = 685.333 us, total = 1.075 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 786 total (1 active), Execution time: mean = 1.424 ms, total = 1.119 s, Queueing time: mean = 72.916 us, max = 323.875 us, min = -0.000 s, total = 57.312 ms GCSServer.deadline_timer.debug_state_event_stats_print - 131 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 209.955 ms, Queueing time: mean = 65.084 us, max = 182.019 us, min = 12.907 us, total = 8.526 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:12:54,023 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 362912 total (1 active) Queueing time: mean = 50.180 us, max = 7.565 ms, min = 1.647 us, total = 18.211 s Execution time: mean = 257.999 us, total = 93.631 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 164955 total (0 active), Execution time: mean = 533.336 us, total = 87.976 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 164955 total (0 active), Execution time: mean = 28.602 us, total = 4.718 s, Queueing time: mean = 107.621 us, max = 7.565 ms, min = 3.372 us, total = 17.753 s event_loop_lag_probe - 31428 total (0 active), Execution time: mean = 29.210 us, total = 918.020 ms, Queueing time: mean = 10.370 us, max = 4.983 ms, min = 1.647 us, total = 325.899 ms GcsTaskManager.GcJobSummary - 1573 total (1 active), Execution time: mean = 11.720 us, total = 18.436 ms, Queueing time: mean = 84.012 us, max = 338.131 us, min = 16.725 us, total = 132.151 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:12:54,023 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38628 total (25 active) Queueing time: mean = 19.703 us, max = 333.684 us, min = 2.330 us, total = 761.082 ms Execution time: mean = 4.807 s, total = 185680.714 s Event stats: event_loop_lag_probe - 31428 total (0 active), Execution time: mean = 31.976 us, total = 1.005 s, Queueing time: mean = 11.782 us, max = 291.078 us, min = 2.330 us, total = 370.277 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1942 total (0 active), Execution time: mean = 43.967 us, total = 85.385 ms, Queueing time: mean = 105.442 us, max = 333.684 us, min = 6.593 us, total = 204.768 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1942 total (24 active), Execution time: mean = 95.612 s, total = 185678.342 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1619 total (0 active), Execution time: mean = 117.190 us, total = 189.731 ms, Queueing time: mean = 111.331 us, max = 293.835 us, min = 16.251 us, total = 180.245 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1619 total (0 active), Execution time: mean = 653.663 us, total = 1.058 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 632.327 us, total = 17.073 ms, Queueing time: mean = 87.982 us, max = 153.556 us, min = 48.361 us, total = 2.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:12:54,023 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31456 total (0 active) Queueing time: mean = 9.915 us, max = 1.423 ms, min = 568.000 ns, total = 311.900 ms Execution time: mean = 26.932 us, total = 847.163 ms Event stats: event_loop_lag_probe - 31430 total (0 active), Execution time: mean = 26.905 us, total = 845.614 ms, Queueing time: mean = 9.864 us, max = 1.423 ms, min = 1.850 us, total = 310.021 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:13:54,024 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1580 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1631 - cumulative published bytes: 1299135 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:13:54,025 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 95275 total (4 active) Queueing time: mean = 68.158 us, max = 279.303 ms, min = -0.000 s, total = 6.494 s Execution time: mean = 245.350 us, total = 23.376 s Event stats: event_loop_lag_probe - 31669 total (0 active), Execution time: mean = 25.464 us, total = 806.408 ms, Queueing time: mean = 10.037 us, max = 13.097 ms, min = 1.994 us, total = 317.852 ms GcsInMemoryStore.Put - 12710 total (0 active), Execution time: mean = 27.466 us, total = 349.095 ms, Queueing time: mean = 200.132 us, max = 278.545 ms, min = 1.649 us, total = 2.544 s RayletLoadPulled - 7918 total (1 active), Execution time: mean = 249.266 us, total = 1.974 s, Queueing time: mean = 75.786 us, max = 472.090 us, min = -0.000 s, total = 600.070 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7915 total (0 active), Execution time: mean = 77.534 us, total = 613.685 ms, Queueing time: mean = 103.649 us, max = 2.744 ms, min = 11.645 us, total = 820.381 ms NodeManagerService.grpc_client.GetResourceLoad - 7915 total (0 active), Execution time: mean = 1.499 ms, total = 11.861 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3174 total (0 active), Execution time: mean = 43.592 us, total = 138.361 ms, Queueing time: mean = 105.048 us, max = 479.211 us, min = 11.443 us, total = 333.424 ms InternalKVGcsService.grpc_server.InternalKVPut - 3174 total (0 active), Execution time: mean = 570.707 us, total = 1.811 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3167 total (0 active), Execution time: mean = 35.963 us, total = 113.896 ms, Queueing time: mean = 117.875 us, max = 1.498 ms, min = 20.329 us, total = 373.309 ms NodeInfoGcsService.grpc_server.CheckAlive - 3167 total (0 active), Execution time: mean = 594.560 us, total = 1.883 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2641 total (1 active), Execution time: mean = 9.612 us, total = 25.385 ms, Queueing time: mean = 76.101 us, max = 391.368 us, min = 10.687 us, total = 200.983 ms HealthCheck - 2637 total (0 active), Execution time: mean = 7.764 us, total = 20.473 ms, Queueing time: mean = 104.129 us, max = 1.431 ms, min = 16.806 us, total = 274.589 ms GcsInMemoryStore.Get - 1649 total (0 active), Execution time: mean = 21.099 us, total = 34.792 ms, Queueing time: mean = 13.525 us, max = 128.508 us, min = 3.701 us, total = 22.302 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1648 total (0 active), Execution time: mean = 58.817 us, total = 96.931 ms, Queueing time: mean = 105.890 us, max = 269.544 us, min = 13.461 us, total = 174.507 ms InternalKVGcsService.grpc_server.InternalKVGet - 1648 total (0 active), Execution time: mean = 616.724 us, total = 1.016 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1580 total (0 active), Execution time: mean = 93.510 us, total = 147.746 ms, Queueing time: mean = 118.223 us, max = 296.469 us, min = 14.122 us, total = 186.793 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1580 total (0 active), Execution time: mean = 686.042 us, total = 1.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 792 total (1 active), Execution time: mean = 1.424 ms, total = 1.128 s, Queueing time: mean = 73.050 us, max = 323.875 us, min = -0.000 s, total = 57.856 ms GCSServer.deadline_timer.debug_state_event_stats_print - 132 total (1 active, 1 running), Execution time: mean = 1.603 ms, total = 211.627 ms, Queueing time: mean = 65.219 us, max = 182.019 us, min = 12.907 us, total = 8.609 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:13:54,025 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 365683 total (1 active) Queueing time: mean = 50.185 us, max = 7.565 ms, min = 1.647 us, total = 18.352 s Execution time: mean = 258.065 us, total = 94.370 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 166215 total (0 active), Execution time: mean = 533.468 us, total = 88.670 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 166215 total (0 active), Execution time: mean = 28.613 us, total = 4.756 s, Queueing time: mean = 107.632 us, max = 7.565 ms, min = 3.372 us, total = 17.890 s event_loop_lag_probe - 31667 total (0 active), Execution time: mean = 29.216 us, total = 925.182 ms, Queueing time: mean = 10.371 us, max = 4.983 ms, min = 1.647 us, total = 328.414 ms GcsTaskManager.GcJobSummary - 1585 total (1 active), Execution time: mean = 11.716 us, total = 18.570 ms, Queueing time: mean = 84.060 us, max = 338.131 us, min = 16.725 us, total = 133.235 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:13:54,025 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 38915 total (25 active) Queueing time: mean = 19.696 us, max = 333.684 us, min = 2.330 us, total = 766.459 ms Execution time: mean = 4.773 s, total = 185740.921 s Event stats: event_loop_lag_probe - 31667 total (0 active), Execution time: mean = 31.987 us, total = 1.013 s, Queueing time: mean = 11.787 us, max = 291.078 us, min = 2.330 us, total = 373.268 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1954 total (0 active), Execution time: mean = 43.992 us, total = 85.960 ms, Queueing time: mean = 105.434 us, max = 333.684 us, min = 6.593 us, total = 206.018 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1954 total (24 active), Execution time: mean = 95.056 s, total = 185738.532 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1631 total (0 active), Execution time: mean = 117.101 us, total = 190.992 ms, Queueing time: mean = 111.208 us, max = 293.835 us, min = 16.251 us, total = 181.380 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1631 total (0 active), Execution time: mean = 653.095 us, total = 1.065 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 632.327 us, total = 17.073 ms, Queueing time: mean = 87.982 us, max = 153.556 us, min = 48.361 us, total = 2.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:13:54,025 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31696 total (0 active) Queueing time: mean = 9.916 us, max = 1.423 ms, min = 568.000 ns, total = 314.286 ms Execution time: mean = 26.929 us, total = 853.536 ms Event stats: event_loop_lag_probe - 31670 total (0 active), Execution time: mean = 26.902 us, total = 851.987 ms, Queueing time: mean = 9.864 us, max = 1.423 ms, min = 1.850 us, total = 312.407 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:14:54,025 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1592 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1644 - cumulative published bytes: 1308997 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:14:54,026 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 95994 total (4 active) Queueing time: mean = 67.954 us, max = 279.303 ms, min = -0.000 s, total = 6.523 s Execution time: mean = 245.340 us, total = 23.551 s Event stats: event_loop_lag_probe - 31909 total (0 active), Execution time: mean = 25.475 us, total = 812.870 ms, Queueing time: mean = 10.033 us, max = 13.097 ms, min = 1.785 us, total = 320.151 ms GcsInMemoryStore.Put - 12806 total (0 active), Execution time: mean = 27.296 us, total = 349.551 ms, Queueing time: mean = 198.956 us, max = 278.545 ms, min = 1.649 us, total = 2.548 s RayletLoadPulled - 7978 total (1 active), Execution time: mean = 249.307 us, total = 1.989 s, Queueing time: mean = 75.865 us, max = 472.090 us, min = -0.000 s, total = 605.251 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 7975 total (0 active), Execution time: mean = 77.525 us, total = 618.265 ms, Queueing time: mean = 103.633 us, max = 2.744 ms, min = 11.645 us, total = 826.471 ms NodeManagerService.grpc_client.GetResourceLoad - 7975 total (0 active), Execution time: mean = 1.499 ms, total = 11.952 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3198 total (0 active), Execution time: mean = 43.549 us, total = 139.270 ms, Queueing time: mean = 104.986 us, max = 479.211 us, min = 11.443 us, total = 335.747 ms InternalKVGcsService.grpc_server.InternalKVPut - 3198 total (0 active), Execution time: mean = 570.630 us, total = 1.825 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3191 total (0 active), Execution time: mean = 35.943 us, total = 114.693 ms, Queueing time: mean = 117.848 us, max = 1.498 ms, min = 20.329 us, total = 376.054 ms NodeInfoGcsService.grpc_server.CheckAlive - 3191 total (0 active), Execution time: mean = 594.423 us, total = 1.897 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2661 total (1 active), Execution time: mean = 9.610 us, total = 25.572 ms, Queueing time: mean = 76.083 us, max = 391.368 us, min = 10.687 us, total = 202.458 ms HealthCheck - 2657 total (0 active), Execution time: mean = 7.763 us, total = 20.628 ms, Queueing time: mean = 104.061 us, max = 1.431 ms, min = 16.806 us, total = 276.491 ms GcsInMemoryStore.Get - 1661 total (0 active), Execution time: mean = 21.070 us, total = 34.997 ms, Queueing time: mean = 13.529 us, max = 128.508 us, min = 3.701 us, total = 22.472 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1660 total (0 active), Execution time: mean = 58.835 us, total = 97.666 ms, Queueing time: mean = 105.943 us, max = 269.544 us, min = 13.461 us, total = 175.865 ms InternalKVGcsService.grpc_server.InternalKVGet - 1660 total (0 active), Execution time: mean = 616.767 us, total = 1.024 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1592 total (0 active), Execution time: mean = 93.441 us, total = 148.758 ms, Queueing time: mean = 118.114 us, max = 296.469 us, min = 14.122 us, total = 188.037 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1592 total (0 active), Execution time: mean = 685.913 us, total = 1.092 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 798 total (1 active), Execution time: mean = 1.425 ms, total = 1.137 s, Queueing time: mean = 72.987 us, max = 323.875 us, min = -0.000 s, total = 58.243 ms GCSServer.deadline_timer.debug_state_event_stats_print - 133 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 213.369 ms, Queueing time: mean = 65.281 us, max = 182.019 us, min = 12.907 us, total = 8.682 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:14:54,026 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 368455 total (1 active) Queueing time: mean = 50.199 us, max = 7.565 ms, min = 1.647 us, total = 18.496 s Execution time: mean = 258.113 us, total = 95.103 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 167475 total (0 active), Execution time: mean = 533.565 us, total = 89.359 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 167475 total (0 active), Execution time: mean = 28.627 us, total = 4.794 s, Queueing time: mean = 107.666 us, max = 7.565 ms, min = 3.372 us, total = 18.031 s event_loop_lag_probe - 31907 total (0 active), Execution time: mean = 29.193 us, total = 931.464 ms, Queueing time: mean = 10.363 us, max = 4.983 ms, min = 1.647 us, total = 330.661 ms GcsTaskManager.GcJobSummary - 1597 total (1 active), Execution time: mean = 11.717 us, total = 18.712 ms, Queueing time: mean = 83.979 us, max = 338.131 us, min = 16.725 us, total = 134.115 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:14:54,027 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39207 total (25 active) Queueing time: mean = 19.698 us, max = 333.684 us, min = 2.330 us, total = 772.291 ms Execution time: mean = 4.739 s, total = 185801.076 s Event stats: event_loop_lag_probe - 31907 total (0 active), Execution time: mean = 31.994 us, total = 1.021 s, Queueing time: mean = 11.791 us, max = 291.078 us, min = 2.330 us, total = 376.203 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1967 total (0 active), Execution time: mean = 43.997 us, total = 86.542 ms, Queueing time: mean = 105.480 us, max = 333.684 us, min = 6.593 us, total = 207.480 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1967 total (24 active), Execution time: mean = 94.458 s, total = 185798.668 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1644 total (0 active), Execution time: mean = 117.086 us, total = 192.490 ms, Queueing time: mean = 111.202 us, max = 293.835 us, min = 16.251 us, total = 182.815 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1644 total (0 active), Execution time: mean = 653.232 us, total = 1.074 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 632.327 us, total = 17.073 ms, Queueing time: mean = 87.982 us, max = 153.556 us, min = 48.361 us, total = 2.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:14:54,027 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 31936 total (0 active) Queueing time: mean = 9.911 us, max = 1.423 ms, min = 568.000 ns, total = 316.533 ms Execution time: mean = 26.918 us, total = 859.647 ms Event stats: event_loop_lag_probe - 31910 total (0 active), Execution time: mean = 26.891 us, total = 858.098 ms, Queueing time: mean = 9.861 us, max = 1.423 ms, min = 1.850 us, total = 314.654 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:15:54,027 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1604 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1656 - cumulative published bytes: 1318825 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:15:54,028 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 96713 total (4 active) Queueing time: mean = 67.727 us, max = 279.303 ms, min = -0.000 s, total = 6.550 s Execution time: mean = 245.271 us, total = 23.721 s Event stats: event_loop_lag_probe - 32149 total (0 active), Execution time: mean = 25.476 us, total = 819.040 ms, Queueing time: mean = 10.029 us, max = 13.097 ms, min = 1.785 us, total = 322.410 ms GcsInMemoryStore.Put - 12902 total (0 active), Execution time: mean = 27.128 us, total = 350.006 ms, Queueing time: mean = 197.827 us, max = 278.545 ms, min = 1.649 us, total = 2.552 s RayletLoadPulled - 8038 total (1 active), Execution time: mean = 249.279 us, total = 2.004 s, Queueing time: mean = 75.889 us, max = 472.090 us, min = -0.000 s, total = 609.997 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8035 total (0 active), Execution time: mean = 77.491 us, total = 622.637 ms, Queueing time: mean = 103.434 us, max = 2.744 ms, min = 11.645 us, total = 831.095 ms NodeManagerService.grpc_client.GetResourceLoad - 8035 total (0 active), Execution time: mean = 1.499 ms, total = 12.041 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3222 total (0 active), Execution time: mean = 43.474 us, total = 140.073 ms, Queueing time: mean = 104.840 us, max = 479.211 us, min = 11.443 us, total = 337.796 ms InternalKVGcsService.grpc_server.InternalKVPut - 3222 total (0 active), Execution time: mean = 570.328 us, total = 1.838 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3215 total (0 active), Execution time: mean = 35.953 us, total = 115.588 ms, Queueing time: mean = 117.869 us, max = 1.498 ms, min = 20.329 us, total = 378.949 ms NodeInfoGcsService.grpc_server.CheckAlive - 3215 total (0 active), Execution time: mean = 594.160 us, total = 1.910 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2681 total (1 active), Execution time: mean = 9.603 us, total = 25.745 ms, Queueing time: mean = 76.070 us, max = 391.368 us, min = 10.687 us, total = 203.944 ms HealthCheck - 2677 total (0 active), Execution time: mean = 7.760 us, total = 20.772 ms, Queueing time: mean = 103.930 us, max = 1.431 ms, min = 16.806 us, total = 278.220 ms GcsInMemoryStore.Get - 1673 total (0 active), Execution time: mean = 21.036 us, total = 35.193 ms, Queueing time: mean = 13.505 us, max = 128.508 us, min = 3.701 us, total = 22.593 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1672 total (0 active), Execution time: mean = 58.793 us, total = 98.302 ms, Queueing time: mean = 105.766 us, max = 269.544 us, min = 13.461 us, total = 176.841 ms InternalKVGcsService.grpc_server.InternalKVGet - 1672 total (0 active), Execution time: mean = 616.187 us, total = 1.030 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1604 total (0 active), Execution time: mean = 93.482 us, total = 149.945 ms, Queueing time: mean = 117.955 us, max = 296.469 us, min = 14.122 us, total = 189.200 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1604 total (0 active), Execution time: mean = 685.665 us, total = 1.100 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 804 total (1 active), Execution time: mean = 1.425 ms, total = 1.146 s, Queueing time: mean = 72.921 us, max = 323.875 us, min = -0.000 s, total = 58.629 ms GCSServer.deadline_timer.debug_state_event_stats_print - 134 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 214.975 ms, Queueing time: mean = 64.959 us, max = 182.019 us, min = 12.907 us, total = 8.704 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:15:54,028 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 371227 total (1 active) Queueing time: mean = 50.210 us, max = 7.565 ms, min = 1.647 us, total = 18.639 s Execution time: mean = 258.122 us, total = 95.822 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 168735 total (0 active), Execution time: mean = 533.581 us, total = 90.034 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 168735 total (0 active), Execution time: mean = 28.633 us, total = 4.831 s, Queueing time: mean = 107.692 us, max = 7.565 ms, min = 3.372 us, total = 18.171 s event_loop_lag_probe - 32147 total (0 active), Execution time: mean = 29.168 us, total = 937.664 ms, Queueing time: mean = 10.354 us, max = 4.983 ms, min = 1.647 us, total = 332.865 ms GcsTaskManager.GcJobSummary - 1609 total (1 active), Execution time: mean = 11.711 us, total = 18.842 ms, Queueing time: mean = 83.945 us, max = 338.131 us, min = 16.725 us, total = 135.068 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:15:54,028 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39495 total (25 active) Queueing time: mean = 19.693 us, max = 333.684 us, min = 2.330 us, total = 777.785 ms Execution time: mean = 4.706 s, total = 185861.229 s Event stats: event_loop_lag_probe - 32147 total (0 active), Execution time: mean = 31.986 us, total = 1.028 s, Queueing time: mean = 11.789 us, max = 291.078 us, min = 2.330 us, total = 378.976 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1979 total (0 active), Execution time: mean = 44.033 us, total = 87.141 ms, Queueing time: mean = 105.512 us, max = 333.684 us, min = 6.593 us, total = 208.807 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1979 total (24 active), Execution time: mean = 93.916 s, total = 185858.804 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1656 total (0 active), Execution time: mean = 117.113 us, total = 193.939 ms, Queueing time: mean = 111.237 us, max = 293.835 us, min = 16.251 us, total = 184.209 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1656 total (0 active), Execution time: mean = 653.006 us, total = 1.081 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 27 total (1 active), Execution time: mean = 632.327 us, total = 17.073 ms, Queueing time: mean = 87.982 us, max = 153.556 us, min = 48.361 us, total = 2.376 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:15:54,029 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32176 total (0 active) Queueing time: mean = 9.908 us, max = 1.423 ms, min = 568.000 ns, total = 318.809 ms Execution time: mean = 26.927 us, total = 866.409 ms Event stats: event_loop_lag_probe - 32150 total (0 active), Execution time: mean = 26.901 us, total = 864.860 ms, Queueing time: mean = 9.858 us, max = 1.423 ms, min = 1.850 us, total = 316.930 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:16:54,029 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1616 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1668 - cumulative published bytes: 1328653 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:16:54,030 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 97432 total (4 active) Queueing time: mean = 67.522 us, max = 279.303 ms, min = -0.000 s, total = 6.579 s Execution time: mean = 245.284 us, total = 23.898 s Event stats: event_loop_lag_probe - 32389 total (0 active), Execution time: mean = 25.487 us, total = 825.483 ms, Queueing time: mean = 10.024 us, max = 13.097 ms, min = 1.785 us, total = 324.662 ms GcsInMemoryStore.Put - 12998 total (0 active), Execution time: mean = 26.968 us, total = 350.524 ms, Queueing time: mean = 196.711 us, max = 278.545 ms, min = 1.649 us, total = 2.557 s RayletLoadPulled - 8098 total (1 active), Execution time: mean = 249.229 us, total = 2.018 s, Queueing time: mean = 75.850 us, max = 472.090 us, min = -0.000 s, total = 614.235 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8095 total (0 active), Execution time: mean = 77.515 us, total = 627.488 ms, Queueing time: mean = 103.376 us, max = 2.744 ms, min = 11.645 us, total = 836.827 ms NodeManagerService.grpc_client.GetResourceLoad - 8095 total (0 active), Execution time: mean = 1.499 ms, total = 12.133 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3246 total (0 active), Execution time: mean = 43.598 us, total = 141.518 ms, Queueing time: mean = 104.869 us, max = 479.211 us, min = 11.443 us, total = 340.405 ms InternalKVGcsService.grpc_server.InternalKVPut - 3246 total (0 active), Execution time: mean = 570.670 us, total = 1.852 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3239 total (0 active), Execution time: mean = 35.999 us, total = 116.601 ms, Queueing time: mean = 117.768 us, max = 1.498 ms, min = 20.329 us, total = 381.450 ms NodeInfoGcsService.grpc_server.CheckAlive - 3239 total (0 active), Execution time: mean = 594.171 us, total = 1.925 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2701 total (1 active), Execution time: mean = 9.597 us, total = 25.922 ms, Queueing time: mean = 76.030 us, max = 391.368 us, min = 10.687 us, total = 205.356 ms HealthCheck - 2697 total (0 active), Execution time: mean = 7.759 us, total = 20.925 ms, Queueing time: mean = 103.901 us, max = 1.431 ms, min = 16.806 us, total = 280.222 ms GcsInMemoryStore.Get - 1685 total (0 active), Execution time: mean = 21.020 us, total = 35.419 ms, Queueing time: mean = 13.498 us, max = 128.508 us, min = 3.701 us, total = 22.744 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1684 total (0 active), Execution time: mean = 58.781 us, total = 98.986 ms, Queueing time: mean = 105.769 us, max = 269.544 us, min = 13.461 us, total = 178.115 ms InternalKVGcsService.grpc_server.InternalKVGet - 1684 total (0 active), Execution time: mean = 616.101 us, total = 1.038 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1616 total (0 active), Execution time: mean = 93.581 us, total = 151.227 ms, Queueing time: mean = 117.939 us, max = 296.469 us, min = 14.122 us, total = 190.589 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1616 total (0 active), Execution time: mean = 685.808 us, total = 1.108 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 810 total (1 active), Execution time: mean = 1.424 ms, total = 1.153 s, Queueing time: mean = 73.012 us, max = 323.875 us, min = -0.000 s, total = 59.139 ms GCSServer.deadline_timer.debug_state_event_stats_print - 135 total (1 active, 1 running), Execution time: mean = 1.605 ms, total = 216.689 ms, Queueing time: mean = 65.104 us, max = 182.019 us, min = 12.907 us, total = 8.789 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:16:54,030 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 373999 total (1 active) Queueing time: mean = 50.233 us, max = 7.565 ms, min = 1.647 us, total = 18.787 s Execution time: mean = 258.219 us, total = 96.574 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 169995 total (0 active), Execution time: mean = 533.776 us, total = 90.739 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 169995 total (0 active), Execution time: mean = 28.654 us, total = 4.871 s, Queueing time: mean = 107.741 us, max = 7.565 ms, min = 3.372 us, total = 18.315 s event_loop_lag_probe - 32387 total (0 active), Execution time: mean = 29.160 us, total = 944.406 ms, Queueing time: mean = 10.353 us, max = 4.983 ms, min = 1.647 us, total = 335.318 ms GcsTaskManager.GcJobSummary - 1621 total (1 active), Execution time: mean = 11.711 us, total = 18.983 ms, Queueing time: mean = 83.937 us, max = 338.131 us, min = 16.725 us, total = 136.062 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:16:54,030 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 39794 total (25 active) Queueing time: mean = 19.694 us, max = 333.684 us, min = 2.330 us, total = 783.684 ms Execution time: mean = 4.747 s, total = 188921.444 s Event stats: event_loop_lag_probe - 32387 total (0 active), Execution time: mean = 32.001 us, total = 1.036 s, Queueing time: mean = 11.794 us, max = 291.078 us, min = 2.330 us, total = 381.982 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 1996 total (0 active), Execution time: mean = 44.033 us, total = 87.890 ms, Queueing time: mean = 105.332 us, max = 333.684 us, min = 6.593 us, total = 210.242 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 1996 total (24 active), Execution time: mean = 94.649 s, total = 188918.999 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1668 total (0 active), Execution time: mean = 117.329 us, total = 195.705 ms, Queueing time: mean = 111.273 us, max = 293.835 us, min = 16.251 us, total = 185.603 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1668 total (0 active), Execution time: mean = 653.331 us, total = 1.090 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 621.458 us, total = 17.401 ms, Queueing time: mean = 87.142 us, max = 153.556 us, min = 48.361 us, total = 2.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:16:54,030 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32415 total (0 active) Queueing time: mean = 9.901 us, max = 1.423 ms, min = 568.000 ns, total = 320.927 ms Execution time: mean = 26.909 us, total = 872.243 ms Event stats: event_loop_lag_probe - 32389 total (0 active), Execution time: mean = 26.882 us, total = 870.694 ms, Queueing time: mean = 9.850 us, max = 1.423 ms, min = 1.850 us, total = 319.048 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:17:54,031 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1628 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1680 - cumulative published bytes: 1338481 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:17:54,032 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 98149 total (4 active) Queueing time: mean = 67.359 us, max = 279.303 ms, min = -0.000 s, total = 6.611 s Execution time: mean = 245.309 us, total = 24.077 s Event stats: event_loop_lag_probe - 32629 total (0 active), Execution time: mean = 25.487 us, total = 831.611 ms, Queueing time: mean = 10.020 us, max = 13.097 ms, min = 1.785 us, total = 326.937 ms GcsInMemoryStore.Put - 13094 total (0 active), Execution time: mean = 26.807 us, total = 351.016 ms, Queueing time: mean = 195.626 us, max = 278.545 ms, min = 1.649 us, total = 2.562 s RayletLoadPulled - 8158 total (1 active), Execution time: mean = 249.371 us, total = 2.034 s, Queueing time: mean = 75.882 us, max = 472.090 us, min = -0.000 s, total = 619.045 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8155 total (0 active), Execution time: mean = 77.522 us, total = 632.192 ms, Queueing time: mean = 103.495 us, max = 2.744 ms, min = 11.645 us, total = 843.999 ms NodeManagerService.grpc_client.GetResourceLoad - 8155 total (0 active), Execution time: mean = 1.499 ms, total = 12.227 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3270 total (0 active), Execution time: mean = 43.578 us, total = 142.499 ms, Queueing time: mean = 104.967 us, max = 479.211 us, min = 11.443 us, total = 343.242 ms InternalKVGcsService.grpc_server.InternalKVPut - 3270 total (0 active), Execution time: mean = 570.725 us, total = 1.866 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3262 total (0 active), Execution time: mean = 35.986 us, total = 117.386 ms, Queueing time: mean = 117.911 us, max = 1.498 ms, min = 20.329 us, total = 384.625 ms NodeInfoGcsService.grpc_server.CheckAlive - 3262 total (0 active), Execution time: mean = 594.268 us, total = 1.939 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2721 total (1 active), Execution time: mean = 9.588 us, total = 26.090 ms, Queueing time: mean = 76.103 us, max = 391.368 us, min = 10.687 us, total = 207.077 ms HealthCheck - 2717 total (0 active), Execution time: mean = 7.762 us, total = 21.090 ms, Queueing time: mean = 104.000 us, max = 1.431 ms, min = 16.806 us, total = 282.569 ms GcsInMemoryStore.Get - 1697 total (0 active), Execution time: mean = 20.998 us, total = 35.634 ms, Queueing time: mean = 13.488 us, max = 128.508 us, min = 3.701 us, total = 22.889 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1696 total (0 active), Execution time: mean = 58.737 us, total = 99.617 ms, Queueing time: mean = 105.747 us, max = 269.544 us, min = 13.461 us, total = 179.346 ms InternalKVGcsService.grpc_server.InternalKVGet - 1696 total (0 active), Execution time: mean = 615.759 us, total = 1.044 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1628 total (0 active), Execution time: mean = 93.687 us, total = 152.523 ms, Queueing time: mean = 117.953 us, max = 296.469 us, min = 14.122 us, total = 192.028 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1628 total (0 active), Execution time: mean = 685.835 us, total = 1.117 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 816 total (1 active), Execution time: mean = 1.424 ms, total = 1.162 s, Queueing time: mean = 73.125 us, max = 323.875 us, min = -0.000 s, total = 59.670 ms GCSServer.deadline_timer.debug_state_event_stats_print - 136 total (1 active, 1 running), Execution time: mean = 1.606 ms, total = 218.462 ms, Queueing time: mean = 65.239 us, max = 182.019 us, min = 12.907 us, total = 8.873 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:17:54,032 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 376771 total (1 active) Queueing time: mean = 50.259 us, max = 7.565 ms, min = 1.647 us, total = 18.936 s Execution time: mean = 258.331 us, total = 97.332 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 171255 total (0 active), Execution time: mean = 533.993 us, total = 91.449 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 171255 total (0 active), Execution time: mean = 28.680 us, total = 4.912 s, Queueing time: mean = 107.798 us, max = 7.565 ms, min = 3.372 us, total = 18.461 s event_loop_lag_probe - 32627 total (0 active), Execution time: mean = 29.175 us, total = 951.882 ms, Queueing time: mean = 10.359 us, max = 4.983 ms, min = 1.647 us, total = 337.985 ms GcsTaskManager.GcJobSummary - 1633 total (1 active), Execution time: mean = 11.704 us, total = 19.113 ms, Queueing time: mean = 83.915 us, max = 338.131 us, min = 16.725 us, total = 137.033 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:17:54,032 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40082 total (25 active) Queueing time: mean = 19.692 us, max = 333.684 us, min = 2.330 us, total = 789.310 ms Execution time: mean = 4.715 s, total = 188981.656 s Event stats: event_loop_lag_probe - 32627 total (0 active), Execution time: mean = 32.006 us, total = 1.044 s, Queueing time: mean = 11.797 us, max = 291.078 us, min = 2.330 us, total = 384.902 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2008 total (0 active), Execution time: mean = 44.135 us, total = 88.623 ms, Queueing time: mean = 105.418 us, max = 333.684 us, min = 6.593 us, total = 211.680 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2008 total (24 active), Execution time: mean = 94.113 s, total = 188979.193 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1680 total (0 active), Execution time: mean = 117.430 us, total = 197.283 ms, Queueing time: mean = 111.232 us, max = 293.835 us, min = 16.251 us, total = 186.870 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1680 total (0 active), Execution time: mean = 653.506 us, total = 1.098 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 621.458 us, total = 17.401 ms, Queueing time: mean = 87.142 us, max = 153.556 us, min = 48.361 us, total = 2.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:17:54,032 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32655 total (0 active) Queueing time: mean = 9.895 us, max = 1.423 ms, min = 568.000 ns, total = 323.135 ms Execution time: mean = 26.895 us, total = 878.269 ms Event stats: event_loop_lag_probe - 32629 total (0 active), Execution time: mean = 26.869 us, total = 876.720 ms, Queueing time: mean = 9.846 us, max = 1.423 ms, min = 1.850 us, total = 321.256 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:18:54,032 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1640 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1692 - cumulative published bytes: 1348309 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:18:54,033 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 98868 total (4 active) Queueing time: mean = 67.129 us, max = 279.303 ms, min = -0.000 s, total = 6.637 s Execution time: mean = 245.184 us, total = 24.241 s Event stats: event_loop_lag_probe - 32869 total (0 active), Execution time: mean = 25.471 us, total = 837.202 ms, Queueing time: mean = 10.008 us, max = 13.097 ms, min = 1.785 us, total = 328.952 ms GcsInMemoryStore.Put - 13190 total (0 active), Execution time: mean = 26.643 us, total = 351.416 ms, Queueing time: mean = 194.483 us, max = 278.545 ms, min = 1.649 us, total = 2.565 s RayletLoadPulled - 8218 total (1 active), Execution time: mean = 249.269 us, total = 2.048 s, Queueing time: mean = 75.858 us, max = 472.090 us, min = -0.000 s, total = 623.400 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8215 total (0 active), Execution time: mean = 77.504 us, total = 636.694 ms, Queueing time: mean = 103.452 us, max = 2.744 ms, min = 11.645 us, total = 849.862 ms NodeManagerService.grpc_client.GetResourceLoad - 8215 total (0 active), Execution time: mean = 1.499 ms, total = 12.312 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3294 total (0 active), Execution time: mean = 43.546 us, total = 143.442 ms, Queueing time: mean = 104.717 us, max = 479.211 us, min = 11.443 us, total = 344.939 ms InternalKVGcsService.grpc_server.InternalKVPut - 3294 total (0 active), Execution time: mean = 570.296 us, total = 1.879 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3286 total (0 active), Execution time: mean = 35.919 us, total = 118.030 ms, Queueing time: mean = 117.719 us, max = 1.498 ms, min = 20.329 us, total = 386.826 ms NodeInfoGcsService.grpc_server.CheckAlive - 3286 total (0 active), Execution time: mean = 593.927 us, total = 1.952 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2741 total (1 active), Execution time: mean = 9.572 us, total = 26.238 ms, Queueing time: mean = 75.967 us, max = 391.368 us, min = 10.687 us, total = 208.227 ms HealthCheck - 2737 total (0 active), Execution time: mean = 7.760 us, total = 21.238 ms, Queueing time: mean = 103.977 us, max = 1.431 ms, min = 16.806 us, total = 284.585 ms GcsInMemoryStore.Get - 1709 total (0 active), Execution time: mean = 20.974 us, total = 35.845 ms, Queueing time: mean = 13.474 us, max = 128.508 us, min = 3.701 us, total = 23.027 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1708 total (0 active), Execution time: mean = 58.665 us, total = 100.200 ms, Queueing time: mean = 105.627 us, max = 269.544 us, min = 13.461 us, total = 180.411 ms InternalKVGcsService.grpc_server.InternalKVGet - 1708 total (0 active), Execution time: mean = 615.320 us, total = 1.051 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1640 total (0 active), Execution time: mean = 93.634 us, total = 153.560 ms, Queueing time: mean = 117.759 us, max = 296.469 us, min = 14.122 us, total = 193.125 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1640 total (0 active), Execution time: mean = 685.475 us, total = 1.124 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 822 total (1 active), Execution time: mean = 1.424 ms, total = 1.171 s, Queueing time: mean = 72.927 us, max = 323.875 us, min = -0.000 s, total = 59.946 ms GCSServer.deadline_timer.debug_state_event_stats_print - 137 total (1 active, 1 running), Execution time: mean = 1.606 ms, total = 220.014 ms, Queueing time: mean = 65.551 us, max = 182.019 us, min = 12.907 us, total = 8.981 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:18:54,033 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 379543 total (1 active) Queueing time: mean = 50.263 us, max = 7.565 ms, min = 1.647 us, total = 19.077 s Execution time: mean = 258.334 us, total = 98.049 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 172515 total (0 active), Execution time: mean = 534.006 us, total = 92.124 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 172515 total (0 active), Execution time: mean = 28.676 us, total = 4.947 s, Queueing time: mean = 107.807 us, max = 7.565 ms, min = 3.372 us, total = 18.598 s event_loop_lag_probe - 32867 total (0 active), Execution time: mean = 29.170 us, total = 958.725 ms, Queueing time: mean = 10.357 us, max = 4.983 ms, min = 1.647 us, total = 340.393 ms GcsTaskManager.GcJobSummary - 1645 total (1 active), Execution time: mean = 11.694 us, total = 19.236 ms, Queueing time: mean = 83.880 us, max = 338.131 us, min = 16.725 us, total = 137.983 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:18:54,034 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40370 total (25 active) Queueing time: mean = 19.691 us, max = 333.684 us, min = 2.330 us, total = 794.939 ms Execution time: mean = 4.683 s, total = 189041.807 s Event stats: event_loop_lag_probe - 32867 total (0 active), Execution time: mean = 32.001 us, total = 1.052 s, Queueing time: mean = 11.796 us, max = 291.078 us, min = 2.330 us, total = 387.703 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2020 total (0 active), Execution time: mean = 44.140 us, total = 89.163 ms, Queueing time: mean = 105.485 us, max = 333.684 us, min = 6.593 us, total = 213.080 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2020 total (24 active), Execution time: mean = 93.584 s, total = 189039.328 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1692 total (0 active), Execution time: mean = 117.305 us, total = 198.479 ms, Queueing time: mean = 111.288 us, max = 293.835 us, min = 16.251 us, total = 188.299 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1692 total (0 active), Execution time: mean = 653.421 us, total = 1.106 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 621.458 us, total = 17.401 ms, Queueing time: mean = 87.142 us, max = 153.556 us, min = 48.361 us, total = 2.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:18:54,034 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 32895 total (0 active) Queueing time: mean = 9.884 us, max = 1.423 ms, min = 568.000 ns, total = 325.119 ms Execution time: mean = 26.863 us, total = 883.663 ms Event stats: event_loop_lag_probe - 32869 total (0 active), Execution time: mean = 26.837 us, total = 882.114 ms, Queueing time: mean = 9.834 us, max = 1.423 ms, min = 1.850 us, total = 323.240 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:19:54,034 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1652 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1704 - cumulative published bytes: 1358137 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:19:54,035 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 99586 total (4 active) Queueing time: mean = 66.926 us, max = 279.303 ms, min = -0.000 s, total = 6.665 s Execution time: mean = 245.155 us, total = 24.414 s Event stats: event_loop_lag_probe - 33108 total (0 active), Execution time: mean = 25.474 us, total = 843.398 ms, Queueing time: mean = 10.004 us, max = 13.097 ms, min = 1.785 us, total = 331.205 ms GcsInMemoryStore.Put - 13286 total (0 active), Execution time: mean = 26.485 us, total = 351.876 ms, Queueing time: mean = 193.383 us, max = 278.545 ms, min = 1.649 us, total = 2.569 s RayletLoadPulled - 8278 total (1 active), Execution time: mean = 249.240 us, total = 2.063 s, Queueing time: mean = 75.867 us, max = 472.090 us, min = -0.000 s, total = 628.028 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8275 total (0 active), Execution time: mean = 77.487 us, total = 641.201 ms, Queueing time: mean = 103.412 us, max = 2.744 ms, min = 11.645 us, total = 855.737 ms NodeManagerService.grpc_client.GetResourceLoad - 8275 total (0 active), Execution time: mean = 1.499 ms, total = 12.404 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3318 total (0 active), Execution time: mean = 43.551 us, total = 144.502 ms, Queueing time: mean = 104.630 us, max = 479.211 us, min = 11.443 us, total = 347.161 ms InternalKVGcsService.grpc_server.InternalKVPut - 3318 total (0 active), Execution time: mean = 570.241 us, total = 1.892 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3310 total (0 active), Execution time: mean = 35.888 us, total = 118.789 ms, Queueing time: mean = 117.657 us, max = 1.498 ms, min = 20.329 us, total = 389.446 ms NodeInfoGcsService.grpc_server.CheckAlive - 3310 total (0 active), Execution time: mean = 593.717 us, total = 1.965 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2761 total (1 active), Execution time: mean = 9.571 us, total = 26.427 ms, Queueing time: mean = 75.937 us, max = 391.368 us, min = 10.687 us, total = 209.661 ms HealthCheck - 2757 total (0 active), Execution time: mean = 7.760 us, total = 21.395 ms, Queueing time: mean = 103.852 us, max = 1.431 ms, min = 16.806 us, total = 286.319 ms GcsInMemoryStore.Get - 1721 total (0 active), Execution time: mean = 20.955 us, total = 36.063 ms, Queueing time: mean = 13.468 us, max = 128.508 us, min = 3.701 us, total = 23.178 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1720 total (0 active), Execution time: mean = 58.618 us, total = 100.823 ms, Queueing time: mean = 105.562 us, max = 269.544 us, min = 13.461 us, total = 181.567 ms InternalKVGcsService.grpc_server.InternalKVGet - 1720 total (0 active), Execution time: mean = 614.894 us, total = 1.058 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1652 total (0 active), Execution time: mean = 93.598 us, total = 154.624 ms, Queueing time: mean = 117.687 us, max = 296.469 us, min = 14.122 us, total = 194.419 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1652 total (0 active), Execution time: mean = 685.300 us, total = 1.132 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 828 total (1 active), Execution time: mean = 1.424 ms, total = 1.179 s, Queueing time: mean = 73.007 us, max = 323.875 us, min = -0.000 s, total = 60.450 ms GCSServer.deadline_timer.debug_state_event_stats_print - 138 total (1 active, 1 running), Execution time: mean = 1.607 ms, total = 221.701 ms, Queueing time: mean = 65.464 us, max = 182.019 us, min = 12.907 us, total = 9.034 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:19:54,035 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 382315 total (1 active) Queueing time: mean = 50.258 us, max = 7.565 ms, min = 1.647 us, total = 19.214 s Execution time: mean = 258.349 us, total = 98.771 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 173775 total (0 active), Execution time: mean = 534.025 us, total = 92.800 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 173775 total (0 active), Execution time: mean = 28.688 us, total = 4.985 s, Queueing time: mean = 107.796 us, max = 7.565 ms, min = 3.372 us, total = 18.732 s event_loop_lag_probe - 33107 total (0 active), Execution time: mean = 29.174 us, total = 965.879 ms, Queueing time: mean = 10.357 us, max = 4.983 ms, min = 1.647 us, total = 342.883 ms GcsTaskManager.GcJobSummary - 1657 total (1 active), Execution time: mean = 11.687 us, total = 19.365 ms, Queueing time: mean = 83.923 us, max = 338.131 us, min = 16.725 us, total = 139.060 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:19:54,035 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40658 total (25 active) Queueing time: mean = 19.692 us, max = 333.684 us, min = 2.330 us, total = 800.617 ms Execution time: mean = 4.651 s, total = 189102.004 s Event stats: event_loop_lag_probe - 33107 total (0 active), Execution time: mean = 31.994 us, total = 1.059 s, Queueing time: mean = 11.794 us, max = 291.078 us, min = 2.330 us, total = 390.465 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2032 total (0 active), Execution time: mean = 44.190 us, total = 89.794 ms, Queueing time: mean = 105.541 us, max = 333.684 us, min = 6.593 us, total = 214.460 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2032 total (24 active), Execution time: mean = 93.061 s, total = 189099.507 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1704 total (0 active), Execution time: mean = 117.300 us, total = 199.879 ms, Queueing time: mean = 111.405 us, max = 293.835 us, min = 16.251 us, total = 189.835 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1704 total (0 active), Execution time: mean = 653.718 us, total = 1.114 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 621.458 us, total = 17.401 ms, Queueing time: mean = 87.142 us, max = 153.556 us, min = 48.361 us, total = 2.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:19:54,035 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33135 total (0 active) Queueing time: mean = 9.880 us, max = 1.423 ms, min = 568.000 ns, total = 327.359 ms Execution time: mean = 26.852 us, total = 889.725 ms Event stats: event_loop_lag_probe - 33109 total (0 active), Execution time: mean = 26.826 us, total = 888.176 ms, Queueing time: mean = 9.831 us, max = 1.423 ms, min = 1.850 us, total = 325.480 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:20:54,036 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1664 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1716 - cumulative published bytes: 1367965 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:20:54,036 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 100305 total (4 active) Queueing time: mean = 66.747 us, max = 279.303 ms, min = -0.000 s, total = 6.695 s Execution time: mean = 245.139 us, total = 24.589 s Event stats: event_loop_lag_probe - 33348 total (0 active), Execution time: mean = 25.477 us, total = 849.610 ms, Queueing time: mean = 10.002 us, max = 13.097 ms, min = 1.785 us, total = 333.534 ms GcsInMemoryStore.Put - 13382 total (0 active), Execution time: mean = 26.356 us, total = 352.691 ms, Queueing time: mean = 192.397 us, max = 278.545 ms, min = 1.649 us, total = 2.575 s RayletLoadPulled - 8338 total (1 active), Execution time: mean = 249.185 us, total = 2.078 s, Queueing time: mean = 75.871 us, max = 472.090 us, min = -0.000 s, total = 632.614 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8335 total (0 active), Execution time: mean = 77.454 us, total = 645.582 ms, Queueing time: mean = 103.372 us, max = 2.744 ms, min = 11.645 us, total = 861.607 ms NodeManagerService.grpc_client.GetResourceLoad - 8335 total (0 active), Execution time: mean = 1.499 ms, total = 12.496 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3342 total (0 active), Execution time: mean = 43.504 us, total = 145.390 ms, Queueing time: mean = 104.702 us, max = 479.211 us, min = 11.443 us, total = 349.913 ms InternalKVGcsService.grpc_server.InternalKVPut - 3342 total (0 active), Execution time: mean = 570.390 us, total = 1.906 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3334 total (0 active), Execution time: mean = 35.906 us, total = 119.711 ms, Queueing time: mean = 117.601 us, max = 1.498 ms, min = 20.329 us, total = 392.081 ms NodeInfoGcsService.grpc_server.CheckAlive - 3334 total (0 active), Execution time: mean = 593.465 us, total = 1.979 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2781 total (1 active), Execution time: mean = 9.575 us, total = 26.629 ms, Queueing time: mean = 75.933 us, max = 391.368 us, min = 10.687 us, total = 211.169 ms HealthCheck - 2777 total (0 active), Execution time: mean = 7.758 us, total = 21.544 ms, Queueing time: mean = 103.758 us, max = 1.431 ms, min = 16.806 us, total = 288.136 ms GcsInMemoryStore.Get - 1733 total (0 active), Execution time: mean = 20.957 us, total = 36.318 ms, Queueing time: mean = 13.459 us, max = 128.508 us, min = 3.701 us, total = 23.324 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1732 total (0 active), Execution time: mean = 58.570 us, total = 101.443 ms, Queueing time: mean = 105.555 us, max = 269.544 us, min = 13.461 us, total = 182.822 ms InternalKVGcsService.grpc_server.InternalKVGet - 1732 total (0 active), Execution time: mean = 614.703 us, total = 1.065 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1664 total (0 active), Execution time: mean = 93.539 us, total = 155.648 ms, Queueing time: mean = 117.773 us, max = 296.469 us, min = 14.122 us, total = 195.974 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1664 total (0 active), Execution time: mean = 685.238 us, total = 1.140 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 834 total (1 active), Execution time: mean = 1.424 ms, total = 1.187 s, Queueing time: mean = 72.913 us, max = 323.875 us, min = -0.000 s, total = 60.810 ms GCSServer.deadline_timer.debug_state_event_stats_print - 139 total (1 active, 1 running), Execution time: mean = 1.607 ms, total = 223.321 ms, Queueing time: mean = 65.159 us, max = 182.019 us, min = 12.907 us, total = 9.057 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:20:54,037 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 385087 total (1 active) Queueing time: mean = 50.267 us, max = 7.565 ms, min = 1.516 us, total = 19.357 s Execution time: mean = 258.396 us, total = 99.505 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 175035 total (0 active), Execution time: mean = 534.129 us, total = 93.491 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 175035 total (0 active), Execution time: mean = 28.688 us, total = 5.021 s, Queueing time: mean = 107.817 us, max = 7.565 ms, min = 3.372 us, total = 18.872 s event_loop_lag_probe - 33347 total (0 active), Execution time: mean = 29.171 us, total = 972.778 ms, Queueing time: mean = 10.355 us, max = 4.983 ms, min = 1.516 us, total = 345.300 ms GcsTaskManager.GcJobSummary - 1669 total (1 active), Execution time: mean = 11.678 us, total = 19.491 ms, Queueing time: mean = 83.992 us, max = 338.131 us, min = 16.725 us, total = 140.182 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:20:54,037 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 40946 total (25 active) Queueing time: mean = 19.683 us, max = 333.684 us, min = 2.330 us, total = 805.959 ms Execution time: mean = 4.620 s, total = 189162.165 s Event stats: event_loop_lag_probe - 33347 total (0 active), Execution time: mean = 31.990 us, total = 1.067 s, Queueing time: mean = 11.790 us, max = 291.078 us, min = 2.330 us, total = 393.166 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2044 total (0 active), Execution time: mean = 44.216 us, total = 90.377 ms, Queueing time: mean = 105.575 us, max = 333.684 us, min = 6.593 us, total = 215.796 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2044 total (24 active), Execution time: mean = 92.544 s, total = 189159.651 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1716 total (0 active), Execution time: mean = 117.263 us, total = 201.222 ms, Queueing time: mean = 111.386 us, max = 293.835 us, min = 16.251 us, total = 191.139 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1716 total (0 active), Execution time: mean = 653.363 us, total = 1.121 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 28 total (1 active), Execution time: mean = 621.458 us, total = 17.401 ms, Queueing time: mean = 87.142 us, max = 153.556 us, min = 48.361 us, total = 2.440 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:20:54,037 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33375 total (0 active) Queueing time: mean = 9.873 us, max = 1.423 ms, min = 568.000 ns, total = 329.518 ms Execution time: mean = 26.847 us, total = 896.009 ms Event stats: event_loop_lag_probe - 33349 total (0 active), Execution time: mean = 26.821 us, total = 894.460 ms, Queueing time: mean = 9.825 us, max = 1.423 ms, min = 1.850 us, total = 327.638 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:21:54,037 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1676 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1729 - cumulative published bytes: 1377827 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:21:54,038 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 101024 total (4 active) Queueing time: mean = 66.578 us, max = 279.303 ms, min = -0.000 s, total = 6.726 s Execution time: mean = 245.149 us, total = 24.766 s Event stats: event_loop_lag_probe - 33588 total (0 active), Execution time: mean = 25.477 us, total = 855.714 ms, Queueing time: mean = 9.999 us, max = 13.097 ms, min = 1.785 us, total = 335.858 ms GcsInMemoryStore.Put - 13478 total (0 active), Execution time: mean = 26.210 us, total = 353.254 ms, Queueing time: mean = 191.370 us, max = 278.545 ms, min = 1.649 us, total = 2.579 s RayletLoadPulled - 8398 total (1 active), Execution time: mean = 249.213 us, total = 2.093 s, Queueing time: mean = 75.907 us, max = 472.090 us, min = -0.000 s, total = 637.469 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8395 total (0 active), Execution time: mean = 77.482 us, total = 650.460 ms, Queueing time: mean = 103.443 us, max = 2.744 ms, min = 11.645 us, total = 868.404 ms NodeManagerService.grpc_client.GetResourceLoad - 8395 total (0 active), Execution time: mean = 1.500 ms, total = 12.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3366 total (0 active), Execution time: mean = 43.505 us, total = 146.438 ms, Queueing time: mean = 104.651 us, max = 479.211 us, min = 11.443 us, total = 352.255 ms InternalKVGcsService.grpc_server.InternalKVPut - 3366 total (0 active), Execution time: mean = 570.598 us, total = 1.921 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3358 total (0 active), Execution time: mean = 35.898 us, total = 120.544 ms, Queueing time: mean = 117.630 us, max = 1.498 ms, min = 20.329 us, total = 395.002 ms NodeInfoGcsService.grpc_server.CheckAlive - 3358 total (0 active), Execution time: mean = 593.459 us, total = 1.993 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2801 total (1 active), Execution time: mean = 9.578 us, total = 26.827 ms, Queueing time: mean = 76.026 us, max = 391.368 us, min = 10.687 us, total = 212.949 ms HealthCheck - 2797 total (0 active), Execution time: mean = 7.763 us, total = 21.712 ms, Queueing time: mean = 103.880 us, max = 1.431 ms, min = 16.806 us, total = 290.552 ms GcsInMemoryStore.Get - 1745 total (0 active), Execution time: mean = 20.934 us, total = 36.530 ms, Queueing time: mean = 13.440 us, max = 128.508 us, min = 3.701 us, total = 23.453 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1744 total (0 active), Execution time: mean = 58.548 us, total = 102.107 ms, Queueing time: mean = 105.406 us, max = 269.544 us, min = 13.461 us, total = 183.829 ms InternalKVGcsService.grpc_server.InternalKVGet - 1744 total (0 active), Execution time: mean = 614.330 us, total = 1.071 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1676 total (0 active), Execution time: mean = 93.654 us, total = 156.965 ms, Queueing time: mean = 117.649 us, max = 296.469 us, min = 14.122 us, total = 197.179 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1676 total (0 active), Execution time: mean = 685.119 us, total = 1.148 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 840 total (1 active), Execution time: mean = 1.424 ms, total = 1.196 s, Queueing time: mean = 72.902 us, max = 323.875 us, min = -0.000 s, total = 61.238 ms GCSServer.deadline_timer.debug_state_event_stats_print - 140 total (1 active, 1 running), Execution time: mean = 1.604 ms, total = 224.590 ms, Queueing time: mean = 65.138 us, max = 182.019 us, min = 12.907 us, total = 9.119 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:21:54,038 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 387859 total (1 active) Queueing time: mean = 50.289 us, max = 7.565 ms, min = 1.516 us, total = 19.505 s Execution time: mean = 258.452 us, total = 100.243 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 176295 total (0 active), Execution time: mean = 534.242 us, total = 94.184 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 176295 total (0 active), Execution time: mean = 28.696 us, total = 5.059 s, Queueing time: mean = 107.862 us, max = 7.565 ms, min = 3.372 us, total = 19.015 s event_loop_lag_probe - 33587 total (0 active), Execution time: mean = 29.183 us, total = 980.155 ms, Queueing time: mean = 10.355 us, max = 4.983 ms, min = 1.516 us, total = 347.791 ms GcsTaskManager.GcJobSummary - 1681 total (1 active), Execution time: mean = 11.677 us, total = 19.629 ms, Queueing time: mean = 84.222 us, max = 338.131 us, min = 16.725 us, total = 141.577 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:21:54,039 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41275 total (25 active) Queueing time: mean = 19.709 us, max = 333.684 us, min = 2.330 us, total = 813.484 ms Execution time: mean = 4.846 s, total = 200022.330 s Event stats: event_loop_lag_probe - 33587 total (0 active), Execution time: mean = 31.990 us, total = 1.074 s, Queueing time: mean = 11.786 us, max = 291.078 us, min = 2.330 us, total = 395.862 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2075 total (0 active), Execution time: mean = 44.086 us, total = 91.478 ms, Queueing time: mean = 105.540 us, max = 333.684 us, min = 6.593 us, total = 218.995 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2075 total (24 active), Execution time: mean = 96.395 s, total = 200019.796 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1729 total (0 active), Execution time: mean = 117.201 us, total = 202.640 ms, Queueing time: mean = 111.423 us, max = 293.835 us, min = 16.251 us, total = 192.651 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1729 total (0 active), Execution time: mean = 653.408 us, total = 1.130 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 629.687 us, total = 18.261 ms, Queueing time: mean = 88.240 us, max = 153.556 us, min = 48.361 us, total = 2.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:21:54,039 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33615 total (0 active) Queueing time: mean = 9.866 us, max = 1.423 ms, min = 568.000 ns, total = 331.639 ms Execution time: mean = 26.826 us, total = 901.754 ms Event stats: event_loop_lag_probe - 33589 total (0 active), Execution time: mean = 26.801 us, total = 900.205 ms, Queueing time: mean = 9.817 us, max = 1.423 ms, min = 1.850 us, total = 329.759 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:22:54,039 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1688 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1742 - cumulative published bytes: 1387689 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:22:54,040 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 101743 total (4 active) Queueing time: mean = 66.360 us, max = 279.303 ms, min = -0.000 s, total = 6.752 s Execution time: mean = 245.045 us, total = 24.932 s Event stats: event_loop_lag_probe - 33828 total (0 active), Execution time: mean = 25.468 us, total = 861.535 ms, Queueing time: mean = 9.994 us, max = 13.097 ms, min = 1.785 us, total = 338.072 ms GcsInMemoryStore.Put - 13574 total (0 active), Execution time: mean = 26.058 us, total = 353.709 ms, Queueing time: mean = 190.355 us, max = 278.545 ms, min = 1.649 us, total = 2.584 s RayletLoadPulled - 8458 total (1 active), Execution time: mean = 249.126 us, total = 2.107 s, Queueing time: mean = 75.868 us, max = 472.090 us, min = -0.000 s, total = 641.688 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8455 total (0 active), Execution time: mean = 77.470 us, total = 655.011 ms, Queueing time: mean = 103.309 us, max = 2.744 ms, min = 11.645 us, total = 873.479 ms NodeManagerService.grpc_client.GetResourceLoad - 8455 total (0 active), Execution time: mean = 1.499 ms, total = 12.677 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3390 total (0 active), Execution time: mean = 43.460 us, total = 147.329 ms, Queueing time: mean = 104.330 us, max = 479.211 us, min = 11.443 us, total = 353.680 ms InternalKVGcsService.grpc_server.InternalKVPut - 3390 total (0 active), Execution time: mean = 569.974 us, total = 1.932 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3382 total (0 active), Execution time: mean = 35.869 us, total = 121.309 ms, Queueing time: mean = 117.555 us, max = 1.498 ms, min = 20.329 us, total = 397.570 ms NodeInfoGcsService.grpc_server.CheckAlive - 3382 total (0 active), Execution time: mean = 593.291 us, total = 2.007 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2821 total (1 active), Execution time: mean = 9.574 us, total = 27.007 ms, Queueing time: mean = 75.958 us, max = 391.368 us, min = 10.687 us, total = 214.277 ms HealthCheck - 2817 total (0 active), Execution time: mean = 7.762 us, total = 21.864 ms, Queueing time: mean = 103.804 us, max = 1.431 ms, min = 16.806 us, total = 292.417 ms GcsInMemoryStore.Get - 1757 total (0 active), Execution time: mean = 20.902 us, total = 36.725 ms, Queueing time: mean = 13.422 us, max = 128.508 us, min = 3.701 us, total = 23.582 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1756 total (0 active), Execution time: mean = 58.458 us, total = 102.652 ms, Queueing time: mean = 105.145 us, max = 269.544 us, min = 13.461 us, total = 184.634 ms InternalKVGcsService.grpc_server.InternalKVGet - 1756 total (0 active), Execution time: mean = 613.638 us, total = 1.078 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1688 total (0 active), Execution time: mean = 93.623 us, total = 158.035 ms, Queueing time: mean = 117.465 us, max = 296.469 us, min = 14.122 us, total = 198.282 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1688 total (0 active), Execution time: mean = 684.793 us, total = 1.156 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 846 total (1 active), Execution time: mean = 1.423 ms, total = 1.204 s, Queueing time: mean = 72.721 us, max = 323.875 us, min = -0.000 s, total = 61.522 ms GCSServer.deadline_timer.debug_state_event_stats_print - 141 total (1 active, 1 running), Execution time: mean = 1.607 ms, total = 226.602 ms, Queueing time: mean = 65.139 us, max = 182.019 us, min = 12.907 us, total = 9.185 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:22:54,040 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 390626 total (1 active) Queueing time: mean = 50.286 us, max = 7.565 ms, min = 1.516 us, total = 19.643 s Execution time: mean = 258.402 us, total = 100.938 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 177553 total (0 active), Execution time: mean = 534.146 us, total = 94.839 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 177553 total (0 active), Execution time: mean = 28.684 us, total = 5.093 s, Queueing time: mean = 107.857 us, max = 7.565 ms, min = 3.372 us, total = 19.150 s event_loop_lag_probe - 33826 total (0 active), Execution time: mean = 29.166 us, total = 986.554 ms, Queueing time: mean = 10.347 us, max = 4.983 ms, min = 1.516 us, total = 350.007 ms GcsTaskManager.GcJobSummary - 1693 total (1 active), Execution time: mean = 11.659 us, total = 19.739 ms, Queueing time: mean = 84.224 us, max = 338.131 us, min = 16.725 us, total = 142.591 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:22:54,040 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41566 total (25 active) Queueing time: mean = 19.697 us, max = 333.684 us, min = 2.330 us, total = 818.736 ms Execution time: mean = 4.814 s, total = 200082.451 s Event stats: event_loop_lag_probe - 33826 total (0 active), Execution time: mean = 31.978 us, total = 1.082 s, Queueing time: mean = 11.781 us, max = 291.078 us, min = 2.330 us, total = 398.514 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2088 total (0 active), Execution time: mean = 44.128 us, total = 92.139 ms, Queueing time: mean = 105.435 us, max = 333.684 us, min = 6.593 us, total = 220.147 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2088 total (24 active), Execution time: mean = 95.824 s, total = 200079.899 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1742 total (0 active), Execution time: mean = 117.369 us, total = 204.457 ms, Queueing time: mean = 111.423 us, max = 293.835 us, min = 16.251 us, total = 194.098 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1742 total (0 active), Execution time: mean = 653.519 us, total = 1.138 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 629.687 us, total = 18.261 ms, Queueing time: mean = 88.240 us, max = 153.556 us, min = 48.361 us, total = 2.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:22:54,041 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 33855 total (0 active) Queueing time: mean = 9.855 us, max = 1.423 ms, min = 568.000 ns, total = 333.630 ms Execution time: mean = 26.797 us, total = 907.200 ms Event stats: event_loop_lag_probe - 33829 total (0 active), Execution time: mean = 26.771 us, total = 905.651 ms, Queueing time: mean = 9.807 us, max = 1.423 ms, min = 1.850 us, total = 331.751 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:23:54,041 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1699 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1753 - cumulative published bytes: 1396698 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:23:54,042 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 102451 total (4 active) Queueing time: mean = 66.170 us, max = 279.303 ms, min = -0.000 s, total = 6.779 s Execution time: mean = 244.977 us, total = 25.098 s Event stats: event_loop_lag_probe - 34068 total (0 active), Execution time: mean = 25.476 us, total = 867.914 ms, Queueing time: mean = 9.993 us, max = 13.097 ms, min = 1.785 us, total = 340.427 ms GcsInMemoryStore.Put - 13668 total (0 active), Execution time: mean = 25.907 us, total = 354.091 ms, Queueing time: mean = 189.325 us, max = 278.545 ms, min = 1.649 us, total = 2.588 s RayletLoadPulled - 8518 total (1 active), Execution time: mean = 249.107 us, total = 2.122 s, Queueing time: mean = 75.906 us, max = 472.090 us, min = -0.000 s, total = 646.565 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8515 total (0 active), Execution time: mean = 77.435 us, total = 659.357 ms, Queueing time: mean = 103.281 us, max = 2.744 ms, min = 11.645 us, total = 879.440 ms NodeManagerService.grpc_client.GetResourceLoad - 8515 total (0 active), Execution time: mean = 1.499 ms, total = 12.765 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3412 total (0 active), Execution time: mean = 43.389 us, total = 148.042 ms, Queueing time: mean = 104.260 us, max = 479.211 us, min = 11.443 us, total = 355.734 ms InternalKVGcsService.grpc_server.InternalKVPut - 3412 total (0 active), Execution time: mean = 569.816 us, total = 1.944 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3406 total (0 active), Execution time: mean = 35.838 us, total = 122.064 ms, Queueing time: mean = 117.437 us, max = 1.498 ms, min = 20.329 us, total = 399.989 ms NodeInfoGcsService.grpc_server.CheckAlive - 3406 total (0 active), Execution time: mean = 593.187 us, total = 2.020 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2841 total (1 active), Execution time: mean = 9.581 us, total = 27.221 ms, Queueing time: mean = 75.942 us, max = 391.368 us, min = 10.687 us, total = 215.752 ms HealthCheck - 2837 total (0 active), Execution time: mean = 7.756 us, total = 22.002 ms, Queueing time: mean = 103.715 us, max = 1.431 ms, min = 16.806 us, total = 294.239 ms GcsInMemoryStore.Get - 1768 total (0 active), Execution time: mean = 20.882 us, total = 36.920 ms, Queueing time: mean = 13.413 us, max = 128.508 us, min = 3.701 us, total = 23.714 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1767 total (0 active), Execution time: mean = 58.405 us, total = 103.202 ms, Queueing time: mean = 105.103 us, max = 269.544 us, min = 13.461 us, total = 185.716 ms InternalKVGcsService.grpc_server.InternalKVGet - 1767 total (0 active), Execution time: mean = 613.213 us, total = 1.084 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1699 total (0 active), Execution time: mean = 93.562 us, total = 158.962 ms, Queueing time: mean = 117.377 us, max = 296.469 us, min = 14.122 us, total = 199.424 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1699 total (0 active), Execution time: mean = 684.874 us, total = 1.164 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 852 total (1 active), Execution time: mean = 1.421 ms, total = 1.211 s, Queueing time: mean = 72.548 us, max = 323.875 us, min = -0.000 s, total = 61.811 ms GCSServer.deadline_timer.debug_state_event_stats_print - 142 total (1 active, 1 running), Execution time: mean = 1.608 ms, total = 228.316 ms, Queueing time: mean = 65.168 us, max = 182.019 us, min = 12.907 us, total = 9.254 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:23:54,042 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 393398 total (1 active) Queueing time: mean = 50.285 us, max = 7.565 ms, min = 1.516 us, total = 19.782 s Execution time: mean = 258.424 us, total = 101.664 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 178813 total (0 active), Execution time: mean = 534.196 us, total = 95.521 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 178813 total (0 active), Execution time: mean = 28.681 us, total = 5.129 s, Queueing time: mean = 107.854 us, max = 7.565 ms, min = 3.372 us, total = 19.286 s event_loop_lag_probe - 34066 total (0 active), Execution time: mean = 29.175 us, total = 993.875 ms, Queueing time: mean = 10.351 us, max = 4.983 ms, min = 1.516 us, total = 352.606 ms GcsTaskManager.GcJobSummary - 1705 total (1 active), Execution time: mean = 11.651 us, total = 19.865 ms, Queueing time: mean = 84.278 us, max = 338.131 us, min = 16.725 us, total = 143.693 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:23:54,042 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 41850 total (25 active) Queueing time: mean = 19.689 us, max = 333.684 us, min = 2.330 us, total = 823.977 ms Execution time: mean = 4.782 s, total = 200137.665 s Event stats: event_loop_lag_probe - 34066 total (0 active), Execution time: mean = 31.981 us, total = 1.089 s, Queueing time: mean = 11.778 us, max = 291.078 us, min = 2.330 us, total = 401.221 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2099 total (0 active), Execution time: mean = 44.161 us, total = 92.694 ms, Queueing time: mean = 105.487 us, max = 333.684 us, min = 6.593 us, total = 221.416 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2099 total (24 active), Execution time: mean = 95.348 s, total = 200135.097 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1753 total (0 active), Execution time: mean = 117.322 us, total = 205.665 ms, Queueing time: mean = 111.445 us, max = 293.835 us, min = 16.251 us, total = 195.364 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1753 total (0 active), Execution time: mean = 653.495 us, total = 1.146 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 629.687 us, total = 18.261 ms, Queueing time: mean = 88.240 us, max = 153.556 us, min = 48.361 us, total = 2.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:23:54,042 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34095 total (0 active) Queueing time: mean = 9.848 us, max = 1.423 ms, min = 568.000 ns, total = 335.767 ms Execution time: mean = 26.778 us, total = 912.985 ms Event stats: event_loop_lag_probe - 34069 total (0 active), Execution time: mean = 26.753 us, total = 911.436 ms, Queueing time: mean = 9.800 us, max = 1.423 ms, min = 1.850 us, total = 333.888 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:24:54,042 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1711 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1765 - cumulative published bytes: 1406526 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:24:54,043 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 103170 total (4 active) Queueing time: mean = 65.935 us, max = 279.303 ms, min = -0.000 s, total = 6.802 s Execution time: mean = 244.847 us, total = 25.261 s Event stats: event_loop_lag_probe - 34308 total (0 active), Execution time: mean = 25.467 us, total = 873.721 ms, Queueing time: mean = 10.000 us, max = 13.097 ms, min = 1.785 us, total = 343.088 ms GcsInMemoryStore.Put - 13764 total (0 active), Execution time: mean = 25.758 us, total = 354.538 ms, Queueing time: mean = 188.277 us, max = 278.545 ms, min = 1.649 us, total = 2.591 s RayletLoadPulled - 8578 total (1 active), Execution time: mean = 248.913 us, total = 2.135 s, Queueing time: mean = 75.792 us, max = 472.090 us, min = -0.000 s, total = 650.140 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8575 total (0 active), Execution time: mean = 77.393 us, total = 663.644 ms, Queueing time: mean = 103.014 us, max = 2.744 ms, min = 11.645 us, total = 883.345 ms NodeManagerService.grpc_client.GetResourceLoad - 8575 total (0 active), Execution time: mean = 1.498 ms, total = 12.849 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3436 total (0 active), Execution time: mean = 43.378 us, total = 149.047 ms, Queueing time: mean = 104.082 us, max = 479.211 us, min = 11.443 us, total = 357.624 ms InternalKVGcsService.grpc_server.InternalKVPut - 3436 total (0 active), Execution time: mean = 569.503 us, total = 1.957 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3430 total (0 active), Execution time: mean = 35.864 us, total = 123.015 ms, Queueing time: mean = 117.286 us, max = 1.498 ms, min = 20.329 us, total = 402.291 ms NodeInfoGcsService.grpc_server.CheckAlive - 3430 total (0 active), Execution time: mean = 592.940 us, total = 2.034 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2861 total (1 active), Execution time: mean = 9.588 us, total = 27.432 ms, Queueing time: mean = 75.813 us, max = 391.368 us, min = 10.687 us, total = 216.902 ms HealthCheck - 2857 total (0 active), Execution time: mean = 7.746 us, total = 22.129 ms, Queueing time: mean = 103.460 us, max = 1.431 ms, min = 16.806 us, total = 295.586 ms GcsInMemoryStore.Get - 1780 total (0 active), Execution time: mean = 20.863 us, total = 37.137 ms, Queueing time: mean = 13.397 us, max = 128.508 us, min = 3.701 us, total = 23.847 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1779 total (0 active), Execution time: mean = 58.391 us, total = 103.878 ms, Queueing time: mean = 104.911 us, max = 269.544 us, min = 13.461 us, total = 186.637 ms InternalKVGcsService.grpc_server.InternalKVGet - 1779 total (0 active), Execution time: mean = 612.855 us, total = 1.090 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1711 total (0 active), Execution time: mean = 93.561 us, total = 160.083 ms, Queueing time: mean = 117.282 us, max = 296.469 us, min = 14.122 us, total = 200.669 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1711 total (0 active), Execution time: mean = 684.615 us, total = 1.171 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 858 total (1 active), Execution time: mean = 1.422 ms, total = 1.220 s, Queueing time: mean = 72.503 us, max = 323.875 us, min = -0.000 s, total = 62.208 ms GCSServer.deadline_timer.debug_state_event_stats_print - 143 total (1 active, 1 running), Execution time: mean = 1.607 ms, total = 229.821 ms, Queueing time: mean = 65.157 us, max = 182.019 us, min = 12.907 us, total = 9.317 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:24:54,044 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 396170 total (1 active) Queueing time: mean = 50.277 us, max = 7.565 ms, min = 1.516 us, total = 19.918 s Execution time: mean = 258.402 us, total = 102.371 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 180073 total (0 active), Execution time: mean = 534.158 us, total = 96.187 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 180073 total (0 active), Execution time: mean = 28.673 us, total = 5.163 s, Queueing time: mean = 107.838 us, max = 7.565 ms, min = 3.372 us, total = 19.419 s event_loop_lag_probe - 34306 total (0 active), Execution time: mean = 29.158 us, total = 1.000 s, Queueing time: mean = 10.342 us, max = 4.983 ms, min = 1.516 us, total = 354.804 ms GcsTaskManager.GcJobSummary - 1717 total (1 active), Execution time: mean = 11.631 us, total = 19.971 ms, Queueing time: mean = 84.208 us, max = 338.131 us, min = 16.625 us, total = 144.584 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:24:54,044 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 42138 total (25 active) Queueing time: mean = 19.676 us, max = 333.684 us, min = 2.330 us, total = 829.115 ms Execution time: mean = 4.751 s, total = 200197.826 s Event stats: event_loop_lag_probe - 34306 total (0 active), Execution time: mean = 31.960 us, total = 1.096 s, Queueing time: mean = 11.769 us, max = 291.078 us, min = 2.330 us, total = 403.743 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2111 total (0 active), Execution time: mean = 44.186 us, total = 93.276 ms, Queueing time: mean = 105.520 us, max = 333.684 us, min = 6.593 us, total = 222.753 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2111 total (24 active), Execution time: mean = 94.834 s, total = 200195.241 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1765 total (0 active), Execution time: mean = 117.242 us, total = 206.932 ms, Queueing time: mean = 111.412 us, max = 293.835 us, min = 16.251 us, total = 196.643 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1765 total (0 active), Execution time: mean = 653.307 us, total = 1.153 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 629.687 us, total = 18.261 ms, Queueing time: mean = 88.240 us, max = 153.556 us, min = 48.361 us, total = 2.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:24:54,044 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34335 total (0 active) Queueing time: mean = 9.837 us, max = 1.423 ms, min = 568.000 ns, total = 337.758 ms Execution time: mean = 26.747 us, total = 918.354 ms Event stats: event_loop_lag_probe - 34309 total (0 active), Execution time: mean = 26.722 us, total = 916.805 ms, Queueing time: mean = 9.790 us, max = 1.423 ms, min = 1.850 us, total = 335.879 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us [2025-01-21 01:25:54,044 I 13457 13457] (gcs_server) gcs_server.cc:245: Gcs Debug state: GcsNodeManager: - RegisterNode request count: 1 - DrainNode request count: 0 - GetAllNodeInfo request count: 34 GcsActorManager: - RegisterActor request count: 0 - CreateActor request count: 0 - GetActorInfo request count: 0 - GetNamedActorInfo request count: 0 - GetAllActorInfo request count: 0 - KillActor request count: 0 - ListNamedActors request count: 0 - Registered actors count: 0 - Destroyed actors count: 0 - Named actors count: 0 - Unresolved actors count: 0 - Pending actors count: 0 - Created actors count: 0 - owners_: 0 - actor_to_register_callbacks_: 0 - actor_to_restart_callbacks_: 0 - actor_to_create_callbacks_: 0 - sorted_destroyed_actor_list_: 0 GcsResourceManager: - GetAllAvailableResources request count: 0 - GetAllTotalResources request count: 0 - GetAllResourceUsage request count: 1723 GcsPlacementGroupManager: - CreatePlacementGroup request count: 0 - RemovePlacementGroup request count: 0 - GetPlacementGroup request count: 0 - GetAllPlacementGroup request count: 0 - WaitPlacementGroupUntilReady request count: 0 - GetNamedPlacementGroup request count: 0 - Scheduling pending placement group count: 0 - Registered placement groups count: 0 - Named placement group count: 0 - Pending placement groups count: 0 - Infeasible placement groups count: 0 Publisher: GCS_JOB_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 256 - current buffered bytes: 256 RAY_LOG_CHANNEL - cumulative published messages: 1777 - cumulative published bytes: 1416354 - current buffered bytes: 819 GCS_NODE_INFO_CHANNEL - cumulative published messages: 1 - cumulative published bytes: 534 - current buffered bytes: 0 [runtime env manager] ID to URIs table: [runtime env manager] URIs reference table: GcsTaskManager: -Total num task events reported: 41 -Total num status task events dropped: 0 -Total num profile events dropped: 0 -Current num of task events stored: 11 -Total num of actor creation tasks: 0 -Total num of actor tasks: 0 -Total num of normal tasks: 10 -Total num of driver tasks: 1 GcsAutoscalerStateManager: - last_seen_autoscaler_state_version_: 0 - last_cluster_resource_state_version_: 0 - pending demands: [2025-01-21 01:25:54,045 I 13457 13457] (gcs_server) gcs_server.cc:843: Main service Event stats: Global stats: 103889 total (4 active) Queueing time: mean = 65.725 us, max = 279.303 ms, min = -0.000 s, total = 6.828 s Execution time: mean = 244.718 us, total = 25.423 s Event stats: event_loop_lag_probe - 34548 total (0 active), Execution time: mean = 25.457 us, total = 879.494 ms, Queueing time: mean = 9.994 us, max = 13.097 ms, min = 1.785 us, total = 345.270 ms GcsInMemoryStore.Put - 13860 total (0 active), Execution time: mean = 25.612 us, total = 354.976 ms, Queueing time: mean = 187.260 us, max = 278.545 ms, min = 1.649 us, total = 2.595 s RayletLoadPulled - 8638 total (1 active), Execution time: mean = 248.715 us, total = 2.148 s, Queueing time: mean = 75.710 us, max = 472.090 us, min = -0.000 s, total = 653.984 ms NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 8635 total (0 active), Execution time: mean = 77.343 us, total = 667.856 ms, Queueing time: mean = 102.884 us, max = 2.744 ms, min = 11.645 us, total = 888.401 ms NodeManagerService.grpc_client.GetResourceLoad - 8635 total (0 active), Execution time: mean = 1.498 ms, total = 12.934 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 3460 total (0 active), Execution time: mean = 43.330 us, total = 149.922 ms, Queueing time: mean = 103.874 us, max = 479.211 us, min = 11.443 us, total = 359.403 ms InternalKVGcsService.grpc_server.InternalKVPut - 3460 total (0 active), Execution time: mean = 569.161 us, total = 1.969 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 3454 total (0 active), Execution time: mean = 35.818 us, total = 123.716 ms, Queueing time: mean = 117.169 us, max = 1.498 ms, min = 20.329 us, total = 404.701 ms NodeInfoGcsService.grpc_server.CheckAlive - 3454 total (0 active), Execution time: mean = 592.728 us, total = 2.047 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s ClusterResourceManager.ResetRemoteNodeView - 2881 total (1 active), Execution time: mean = 9.588 us, total = 27.623 ms, Queueing time: mean = 75.844 us, max = 391.368 us, min = 10.687 us, total = 218.505 ms HealthCheck - 2877 total (0 active), Execution time: mean = 7.744 us, total = 22.278 ms, Queueing time: mean = 103.439 us, max = 1.431 ms, min = 16.806 us, total = 297.593 ms GcsInMemoryStore.Get - 1792 total (0 active), Execution time: mean = 20.841 us, total = 37.348 ms, Queueing time: mean = 13.386 us, max = 128.508 us, min = 3.701 us, total = 23.987 ms InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 1791 total (0 active), Execution time: mean = 58.324 us, total = 104.458 ms, Queueing time: mean = 104.778 us, max = 269.544 us, min = 13.461 us, total = 187.658 ms InternalKVGcsService.grpc_server.InternalKVGet - 1791 total (0 active), Execution time: mean = 612.567 us, total = 1.097 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 1723 total (0 active), Execution time: mean = 93.477 us, total = 161.061 ms, Queueing time: mean = 117.150 us, max = 296.469 us, min = 14.122 us, total = 201.850 ms NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 1723 total (0 active), Execution time: mean = 684.362 us, total = 1.179 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GCSServer.deadline_timer.debug_state_dump - 864 total (1 active), Execution time: mean = 1.421 ms, total = 1.228 s, Queueing time: mean = 72.455 us, max = 323.875 us, min = -0.000 s, total = 62.601 ms GCSServer.deadline_timer.debug_state_event_stats_print - 144 total (1 active, 1 running), Execution time: mean = 1.607 ms, total = 231.452 ms, Queueing time: mean = 64.850 us, max = 182.019 us, min = 12.907 us, total = 9.338 ms NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 626.288 us, total = 21.294 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 38.829 us, total = 1.320 ms, Queueing time: mean = 128.920 us, max = 189.981 us, min = 20.090 us, total = 4.383 ms WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 639.802 us, total = 13.436 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 30.627 us, total = 643.161 us, Queueing time: mean = 132.369 us, max = 239.162 us, min = 26.759 us, total = 2.780 ms GcsResourceManager::Update - 12 total (0 active), Execution time: mean = 111.572 us, total = 1.339 ms, Queueing time: mean = 139.823 us, max = 191.304 us, min = 21.005 us, total = 1.678 ms GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 16.446 us, total = 98.675 us, Queueing time: mean = 76.435 us, max = 101.411 us, min = 14.900 us, total = 458.608 us NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.973 ms, total = 9.867 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 857.228 us, total = 4.286 ms, Queueing time: mean = 337.070 us, max = 1.136 ms, min = 109.531 us, total = 1.685 ms PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 329.116 us, total = 1.316 ms, Queueing time: mean = 141.901 ms, max = 279.303 ms, min = 4.231 ms, total = 567.605 ms GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 23.668 us, total = 23.668 us, Queueing time: mean = 13.279 us, max = 13.279 us, min = 13.279 us, total = 13.279 us NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 1.281 ms, total = 1.281 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 906.560 us, total = 906.560 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 16.051 us, total = 16.051 us, Queueing time: mean = 10.229 us, max = 10.229 us, min = 10.229 us, total = 10.229 us GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 17.206 us, total = 17.206 us, Queueing time: mean = 807.000 ns, max = 807.000 ns, min = 807.000 ns, total = 807.000 ns JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 801.197 us, total = 801.197 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 677.583 us, total = 677.583 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 79.707 us, total = 79.707 us, Queueing time: mean = 128.483 us, max = 128.483 us, min = 128.483 us, total = 128.483 us JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 1.012 ms, total = 1.012 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 682.058 us, total = 682.058 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 24.558 us, total = 24.558 us, Queueing time: mean = 134.773 us, max = 134.773 us, min = 134.773 us, total = 134.773 us InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 35.149 us, total = 35.149 us, Queueing time: mean = 48.922 us, max = 48.922 us, min = 48.922 us, total = 48.922 us JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.080 us, total = 58.080 us, Queueing time: mean = 154.830 us, max = 154.830 us, min = 154.830 us, total = 154.830 us InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 493.698 us, total = 493.698 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 212.378 us, total = 212.378 us, Queueing time: mean = 124.333 us, max = 124.333 us, min = 124.333 us, total = 124.333 us JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 153.988 us, total = 153.988 us, Queueing time: mean = 131.439 us, max = 131.439 us, min = 131.439 us, total = 131.439 us InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 32.554 us, total = 32.554 us, Queueing time: mean = 44.202 us, max = 44.202 us, min = 44.202 us, total = 44.202 us [2025-01-21 01:25:54,045 I 13457 13457] (gcs_server) gcs_server.cc:847: task_io_context Event stats: Global stats: 398942 total (1 active) Queueing time: mean = 50.264 us, max = 7.565 ms, min = 1.516 us, total = 20.053 s Execution time: mean = 258.422 us, total = 103.096 s Event stats: TaskInfoGcsService.grpc_server.AddTaskEventData - 181333 total (0 active), Execution time: mean = 534.200 us, total = 96.868 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 181333 total (0 active), Execution time: mean = 28.674 us, total = 5.200 s, Queueing time: mean = 107.811 us, max = 7.565 ms, min = 3.372 us, total = 19.550 s event_loop_lag_probe - 34546 total (0 active), Execution time: mean = 29.171 us, total = 1.008 s, Queueing time: mean = 10.343 us, max = 4.983 ms, min = 1.516 us, total = 357.319 ms GcsTaskManager.GcJobSummary - 1729 total (1 active), Execution time: mean = 11.614 us, total = 20.081 ms, Queueing time: mean = 84.162 us, max = 338.131 us, min = 16.625 us, total = 145.517 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 8.636 us, total = 8.636 us, Queueing time: mean = 99.989 us, max = 99.989 us, min = 99.989 us, total = 99.989 us [2025-01-21 01:25:54,045 I 13457 13457] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: Global stats: 42426 total (25 active) Queueing time: mean = 19.674 us, max = 333.684 us, min = 2.330 us, total = 834.668 ms Execution time: mean = 4.720 s, total = 200257.993 s Event stats: event_loop_lag_probe - 34546 total (0 active), Execution time: mean = 31.958 us, total = 1.104 s, Queueing time: mean = 11.766 us, max = 291.078 us, min = 2.330 us, total = 406.472 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 2123 total (0 active), Execution time: mean = 44.217 us, total = 93.872 ms, Queueing time: mean = 105.508 us, max = 333.684 us, min = 6.593 us, total = 223.994 ms InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 2123 total (24 active), Execution time: mean = 94.327 s, total = 200255.391 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 1777 total (0 active), Execution time: mean = 117.228 us, total = 208.314 ms, Queueing time: mean = 111.551 us, max = 293.835 us, min = 16.251 us, total = 198.226 ms InternalPubSubGcsService.grpc_server.GcsPublish - 1777 total (0 active), Execution time: mean = 653.457 us, total = 1.161 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s Publisher.CheckDeadSubscribers - 29 total (1 active), Execution time: mean = 629.687 us, total = 18.261 ms, Queueing time: mean = 88.240 us, max = 153.556 us, min = 48.361 us, total = 2.559 ms InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 601.626 us, total = 15.041 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 70.883 us, total = 1.772 ms, Queueing time: mean = 132.367 us, max = 300.767 us, min = 26.234 us, total = 3.309 ms PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 18.516 us, total = 18.516 us, Queueing time: mean = 108.109 us, max = 108.109 us, min = 108.109 us, total = 108.109 us [2025-01-21 01:25:54,046 I 13457 13457] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: Global stats: 34575 total (0 active) Queueing time: mean = 9.829 us, max = 1.423 ms, min = 568.000 ns, total = 339.826 ms Execution time: mean = 26.720 us, total = 923.849 ms Event stats: event_loop_lag_probe - 34549 total (0 active), Execution time: mean = 26.695 us, total = 922.300 ms, Queueing time: mean = 9.782 us, max = 1.423 ms, min = 1.850 us, total = 337.947 ms RaySyncer.BroadcastMessage - 12 total (0 active), Execution time: mean = 27.243 us, total = 326.917 us, Queueing time: mean = 1.158 us, max = 1.836 us, min = 568.000 ns, total = 13.895 us - 12 total (0 active), Execution time: mean = 101.657 us, total = 1.220 ms, Queueing time: mean = 134.514 us, max = 163.613 us, min = 33.336 us, total = 1.614 ms RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.110 us, total = 2.220 us, Queueing time: mean = 125.562 us, max = 127.563 us, min = 123.561 us, total = 251.124 us