|
[2025-01-21 05:58:18,764 I 20340 20340] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
[2025-01-21 05:58:18,765 I 20340 20340] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. |
|
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for GCS |
|
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE |
|
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR |
|
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB |
|
[2025-01-21 05:58:18,769 I 20340 20340] (gcs_server) event.cc:324: Set ray event level to warning |
|
[2025-01-21 05:58:18,770 I 20340 20340] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:42: Loading job table data. |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:54: Loading node table data. |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:80: Loading actor table data. |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 9f6a3ba5c12bdaae5991e0c6ce9a10b1c06e51056583b7869135876e |
|
[2025-01-21 05:58:18,771 I 20340 20340] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 |
|
[2025-01-21 05:58:18,773 I 20340 20340] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 58630. |
|
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 0 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 0 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 0 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 0 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 0 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 0 |
|
-Total num of driver tasks: 0 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 23 total (4 active) |
|
Queueing time: mean = 82.234 ms, max = 209.811 ms, min = 1.348 us, total = 1.891 s |
|
Execution time: mean = 9.126 ms, total = 209.887 ms |
|
Event stats: |
|
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 23.315 ms, total = 209.831 ms, Queueing time: mean = 162.795 ms, max = 209.465 ms, min = 1.348 us, total = 1.465 s |
|
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 5.874 us, total = 29.369 us, Queueing time: mean = 36.556 us, max = 39.816 us, min = 33.188 us, total = 182.779 us |
|
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 1.955 us, total = 7.820 us, Queueing time: mean = 104.882 ms, max = 209.811 ms, min = 209.718 ms, total = 419.529 ms |
|
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 5.213 us, total = 10.426 us, Queueing time: mean = 3.260 ms, max = 6.396 ms, min = 124.659 us, total = 6.520 ms |
|
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 8.936 us, total = 8.936 us, Queueing time: mean = 2.067 us, max = 2.067 us, min = 2.067 us, total = 2.067 us |
|
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 4 total (1 active) |
|
Queueing time: mean = 110.391 us, max = 389.975 us, min = 12.503 us, total = 441.565 us |
|
Execution time: mean = 70.540 us, total = 282.161 us |
|
Event stats: |
|
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 137.157 us, total = 274.315 us, Queueing time: mean = 214.531 us, max = 389.975 us, min = 39.087 us, total = 429.062 us |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us |
|
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 4 total (1 active) |
|
Queueing time: mean = 1.141 ms, max = 2.957 ms, min = 113.015 us, total = 4.564 ms |
|
Execution time: mean = 53.532 us, total = 214.128 us |
|
Event stats: |
|
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 79.139 us, total = 158.278 us, Queueing time: mean = 1.535 ms, max = 2.957 ms, min = 113.015 us, total = 3.070 ms |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 05:58:18,981 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 4 total (0 active) |
|
Queueing time: mean = 303.213 us, max = 1.158 ms, min = 12.186 us, total = 1.213 ms |
|
Execution time: mean = 47.383 us, total = 189.532 us |
|
Event stats: |
|
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 94.208 us, total = 188.416 us, Queueing time: mean = 593.180 us, max = 1.158 ms, min = 28.000 us, total = 1.186 ms |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us |
|
|
|
|
|
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
[2025-01-21 05:58:21,131 I 20340 20340] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c registered, will try to reschedule all the infeasible placement groups. |
|
[2025-01-21 05:58:21,136 I 20340 20422] (gcs_server) ray_syncer.cc:377: Get connection node_id=331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
[2025-01-21 05:58:21,922 I 20340 20340] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 20269 |
|
[2025-01-21 05:58:21,922 I 20340 20340] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 20269 |
|
[2025-01-21 05:58:28,780 W 20340 20363] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses |
|
[2025-01-21 05:59:18,981 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 1 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 34 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 12 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
RAY_LOG_CHANNEL |
|
- cumulative published messages: 13 |
|
- cumulative published bytes: 12345 |
|
- current buffered bytes: 818 |
|
GCS_NODE_INFO_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 536 |
|
- current buffered bytes: 0 |
|
GCS_JOB_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 283 |
|
- current buffered bytes: 283 |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 3 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 2 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 1 |
|
-Total num of driver tasks: 1 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 1117 total (4 active) |
|
Queueing time: mean = 1.770 ms, max = 209.811 ms, min = 740.000 ns, total = 1.977 s |
|
Execution time: mean = 436.269 us, total = 487.313 ms |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 21.988 us, total = 5.321 ms, Queueing time: mean = 35.197 us, max = 6.396 ms, min = 2.670 us, total = 8.518 ms |
|
GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.478 ms, total = 211.380 ms, Queueing time: mean = 10.282 ms, max = 209.465 ms, min = 1.325 us, total = 1.470 s |
|
GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 16.750 us, total = 1.206 ms, Queueing time: mean = 21.673 us, max = 158.434 us, min = 2.067 us, total = 1.560 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 834.058 us, total = 59.218 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 33.602 us, total = 2.386 ms, Queueing time: mean = 433.801 us, max = 8.337 ms, min = 11.662 us, total = 30.800 ms |
|
RayletLoadPulled - 60 total (1 active), Execution time: mean = 241.451 us, total = 14.487 ms, Queueing time: mean = 63.665 us, max = 183.264 us, min = 16.365 us, total = 3.820 ms |
|
NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.499 ms, total = 85.458 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 73.211 us, total = 4.173 ms, Queueing time: mean = 101.909 us, max = 178.023 us, min = 19.180 us, total = 5.809 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 32.339 us, total = 1.261 ms, Queueing time: mean = 79.184 us, max = 203.571 us, min = 8.800 us, total = 3.088 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 497.287 us, total = 19.394 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 30.708 us, total = 736.999 us, Queueing time: mean = 107.406 us, max = 207.471 us, min = 15.170 us, total = 2.578 ms |
|
NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 521.698 us, total = 12.521 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 9.595 us, total = 191.895 us, Queueing time: mean = 146.339 us, max = 1.634 ms, min = 30.256 us, total = 2.927 ms |
|
HealthCheck - 18 total (0 active), Execution time: mean = 8.213 us, total = 147.828 us, Queueing time: mean = 110.846 us, max = 204.653 us, min = 22.112 us, total = 1.995 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 644.168 us, total = 7.730 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 81.180 us, total = 974.157 us, Queueing time: mean = 105.073 us, max = 201.135 us, min = 20.386 us, total = 1.261 ms |
|
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us |
|
GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.140 ms, total = 6.837 ms, Queueing time: mean = 58.245 us, max = 119.431 us, min = 20.387 us, total = 349.473 us |
|
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us |
|
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms |
|
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us |
|
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us |
|
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us |
|
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us |
|
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us |
|
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us |
|
GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns |
|
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us |
|
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us |
|
|
|
|
|
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 2692 total (1 active) |
|
Queueing time: mean = 49.443 us, max = 498.514 us, min = 2.501 us, total = 133.102 ms |
|
Execution time: mean = 250.657 us, total = 674.768 ms |
|
Event stats: |
|
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1218 total (0 active), Execution time: mean = 24.867 us, total = 30.288 ms, Queueing time: mean = 106.284 us, max = 498.514 us, min = 6.180 us, total = 129.454 ms |
|
TaskInfoGcsService.grpc_server.AddTaskEventData - 1218 total (0 active), Execution time: mean = 523.411 us, total = 637.515 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 28.218 us, total = 6.829 ms, Queueing time: mean = 11.157 us, max = 389.975 us, min = 2.501 us, total = 2.700 ms |
|
GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 9.916 us, total = 128.907 us, Queueing time: mean = 71.929 us, max = 155.412 us, min = 40.036 us, total = 935.078 us |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us |
|
|
|
|
|
[2025-01-21 05:59:18,982 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 394 total (25 active) |
|
Queueing time: mean = 33.171 us, max = 2.957 ms, min = 3.295 us, total = 13.069 ms |
|
Execution time: mean = 136.099 ms, total = 53.623 s |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 24.367 us, total = 5.897 ms, Queueing time: mean = 21.342 us, max = 2.957 ms, min = 3.295 us, total = 5.165 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 37 total (24 active), Execution time: mean = 1.448 s, total = 53.588 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 37 total (0 active), Execution time: mean = 28.725 us, total = 1.063 ms, Queueing time: mean = 89.387 us, max = 239.158 us, min = 15.207 us, total = 3.307 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 13 total (0 active), Execution time: mean = 112.098 us, total = 1.457 ms, Queueing time: mean = 100.760 us, max = 175.467 us, min = 19.350 us, total = 1.310 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish - 13 total (0 active), Execution time: mean = 590.116 us, total = 7.672 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 05:59:18,983 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 248 total (0 active) |
|
Queueing time: mean = 14.569 us, max = 1.158 ms, min = 692.000 ns, total = 3.613 ms |
|
Execution time: mean = 27.721 us, total = 6.875 ms |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 27.609 us, total = 6.681 ms, Queueing time: mean = 14.627 us, max = 1.158 ms, min = 5.068 us, total = 3.540 ms |
|
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us |
|
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us |
|
|
|
|
|
[2025-01-21 06:00:18,983 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 1 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 34 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 24 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
RAY_LOG_CHANNEL |
|
- cumulative published messages: 26 |
|
- cumulative published bytes: 22194 |
|
- current buffered bytes: 818 |
|
GCS_NODE_INFO_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 536 |
|
- current buffered bytes: 0 |
|
GCS_JOB_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 283 |
|
- current buffered bytes: 283 |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 3 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 2 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 1 |
|
-Total num of driver tasks: 1 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 1836 total (4 active) |
|
Queueing time: mean = 1.089 ms, max = 209.811 ms, min = 740.000 ns, total = 1.999 s |
|
Execution time: mean = 343.881 us, total = 631.365 ms |
|
Event stats: |
|
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 22.394 us, total = 10.794 ms, Queueing time: mean = 21.955 us, max = 6.396 ms, min = 2.670 us, total = 10.582 ms |
|
GcsInMemoryStore.Put - 239 total (0 active), Execution time: mean = 886.425 us, total = 211.856 ms, Queueing time: mean = 6.168 ms, max = 209.465 ms, min = 1.325 us, total = 1.474 s |
|
RayletLoadPulled - 120 total (1 active), Execution time: mean = 236.316 us, total = 28.358 ms, Queueing time: mean = 61.931 us, max = 183.264 us, min = 16.365 us, total = 7.432 ms |
|
NodeManagerService.grpc_client.GetResourceLoad - 117 total (0 active), Execution time: mean = 1.403 ms, total = 164.165 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 117 total (0 active), Execution time: mean = 69.642 us, total = 8.148 ms, Queueing time: mean = 89.966 us, max = 178.023 us, min = 19.180 us, total = 10.526 ms |
|
GcsInMemoryStore.Get - 84 total (0 active), Execution time: mean = 17.022 us, total = 1.430 ms, Queueing time: mean = 20.124 us, max = 158.434 us, min = 2.067 us, total = 1.690 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet - 83 total (0 active), Execution time: mean = 767.781 us, total = 63.726 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 83 total (0 active), Execution time: mean = 35.450 us, total = 2.942 ms, Queueing time: mean = 377.364 us, max = 8.337 ms, min = 11.662 us, total = 31.321 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 63 total (0 active), Execution time: mean = 31.098 us, total = 1.959 ms, Queueing time: mean = 67.742 us, max = 203.571 us, min = 8.800 us, total = 4.268 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut - 63 total (0 active), Execution time: mean = 441.874 us, total = 27.838 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 48 total (0 active), Execution time: mean = 28.084 us, total = 1.348 ms, Queueing time: mean = 93.026 us, max = 207.471 us, min = 15.170 us, total = 4.465 ms |
|
NodeInfoGcsService.grpc_server.CheckAlive - 48 total (0 active), Execution time: mean = 487.566 us, total = 23.403 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
ClusterResourceManager.ResetRemoteNodeView - 40 total (1 active), Execution time: mean = 9.161 us, total = 366.445 us, Queueing time: mean = 105.860 us, max = 1.634 ms, min = 26.287 us, total = 4.234 ms |
|
HealthCheck - 38 total (0 active), Execution time: mean = 7.210 us, total = 273.970 us, Queueing time: mean = 96.160 us, max = 204.653 us, min = 20.297 us, total = 3.654 ms |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 24 total (0 active), Execution time: mean = 76.622 us, total = 1.839 ms, Queueing time: mean = 87.892 us, max = 201.135 us, min = 20.386 us, total = 2.109 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 24 total (0 active), Execution time: mean = 551.027 us, total = 13.225 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GCSServer.deadline_timer.debug_state_dump - 12 total (1 active), Execution time: mean = 1.208 ms, total = 14.500 ms, Queueing time: mean = 44.701 us, max = 119.431 us, min = 16.811 us, total = 536.407 us |
|
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us |
|
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us |
|
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms |
|
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us |
|
GCSServer.deadline_timer.debug_state_event_stats_print - 2 total (1 active, 1 running), Execution time: mean = 652.030 us, total = 1.304 ms, Queueing time: mean = 38.792 us, max = 77.585 us, min = 77.585 us, total = 77.585 us |
|
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us |
|
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us |
|
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns |
|
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us |
|
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us |
|
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us |
|
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us |
|
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us |
|
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 5464 total (1 active) |
|
Queueing time: mean = 42.465 us, max = 606.639 us, min = 2.131 us, total = 232.028 ms |
|
Execution time: mean = 219.054 us, total = 1.197 s |
|
Event stats: |
|
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 2478 total (0 active), Execution time: mean = 21.459 us, total = 53.175 ms, Queueing time: mean = 90.955 us, max = 498.514 us, min = 5.172 us, total = 225.386 ms |
|
TaskInfoGcsService.grpc_server.AddTaskEventData - 2478 total (0 active), Execution time: mean = 456.554 us, total = 1.131 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 25.227 us, total = 12.159 ms, Queueing time: mean = 9.330 us, max = 389.975 us, min = 2.131 us, total = 4.497 ms |
|
GcsTaskManager.GcJobSummary - 25 total (1 active), Execution time: mean = 9.176 us, total = 229.403 us, Queueing time: mean = 85.311 us, max = 606.639 us, min = 32.578 us, total = 2.133 ms |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us |
|
|
|
|
|
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 686 total (25 active) |
|
Queueing time: mean = 23.604 us, max = 2.957 ms, min = 3.295 us, total = 16.192 ms |
|
Execution time: mean = 165.942 ms, total = 113.836 s |
|
Event stats: |
|
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 23.776 us, total = 11.460 ms, Queueing time: mean = 14.910 us, max = 2.957 ms, min = 3.295 us, total = 7.186 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 50 total (24 active), Execution time: mean = 2.276 s, total = 113.789 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 50 total (0 active), Execution time: mean = 29.893 us, total = 1.495 ms, Queueing time: mean = 77.151 us, max = 239.158 us, min = 13.433 us, total = 3.858 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 26 total (0 active), Execution time: mean = 94.267 us, total = 2.451 ms, Queueing time: mean = 71.579 us, max = 175.467 us, min = 17.549 us, total = 1.861 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish - 26 total (0 active), Execution time: mean = 485.686 us, total = 12.628 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:00:18,984 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 488 total (0 active) |
|
Queueing time: mean = 11.901 us, max = 1.158 ms, min = 692.000 ns, total = 5.808 ms |
|
Execution time: mean = 26.169 us, total = 12.771 ms |
|
Event stats: |
|
event_loop_lag_probe - 482 total (0 active), Execution time: mean = 26.093 us, total = 12.577 ms, Queueing time: mean = 11.898 us, max = 1.158 ms, min = 4.729 us, total = 5.735 ms |
|
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us |
|
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us |
|
|
|
|
|
[2025-01-21 06:01:18,985 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 1 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 34 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 36 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
RAY_LOG_CHANNEL |
|
- cumulative published messages: 39 |
|
- cumulative published bytes: 32043 |
|
- current buffered bytes: 818 |
|
GCS_NODE_INFO_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 536 |
|
- current buffered bytes: 0 |
|
GCS_JOB_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 283 |
|
- current buffered bytes: 283 |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 3 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 2 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 1 |
|
-Total num of driver tasks: 1 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 06:01:18,985 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 2555 total (4 active) |
|
Queueing time: mean = 794.373 us, max = 209.811 ms, min = 740.000 ns, total = 2.030 s |
|
Execution time: mean = 317.087 us, total = 810.158 ms |
|
Event stats: |
|
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 23.405 us, total = 16.898 ms, Queueing time: mean = 17.921 us, max = 6.396 ms, min = 2.670 us, total = 12.939 ms |
|
GcsInMemoryStore.Put - 335 total (0 active), Execution time: mean = 634.154 us, total = 212.442 ms, Queueing time: mean = 4.413 ms, max = 209.465 ms, min = 1.325 us, total = 1.478 s |
|
RayletLoadPulled - 180 total (1 active), Execution time: mean = 245.093 us, total = 44.117 ms, Queueing time: mean = 68.265 us, max = 183.264 us, min = 16.365 us, total = 12.288 ms |
|
NodeManagerService.grpc_client.GetResourceLoad - 177 total (0 active), Execution time: mean = 1.462 ms, total = 258.807 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 177 total (0 active), Execution time: mean = 72.124 us, total = 12.766 ms, Queueing time: mean = 99.180 us, max = 183.295 us, min = 19.180 us, total = 17.555 ms |
|
GcsInMemoryStore.Get - 96 total (0 active), Execution time: mean = 18.975 us, total = 1.822 ms, Queueing time: mean = 19.112 us, max = 158.434 us, min = 2.067 us, total = 1.835 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet - 95 total (0 active), Execution time: mean = 746.310 us, total = 70.899 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 95 total (0 active), Execution time: mean = 37.327 us, total = 3.546 ms, Queueing time: mean = 341.891 us, max = 8.337 ms, min = 11.662 us, total = 32.480 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 87 total (0 active), Execution time: mean = 34.112 us, total = 2.968 ms, Queueing time: mean = 76.665 us, max = 203.571 us, min = 8.800 us, total = 6.670 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut - 87 total (0 active), Execution time: mean = 473.275 us, total = 41.175 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 72 total (0 active), Execution time: mean = 30.253 us, total = 2.178 ms, Queueing time: mean = 101.638 us, max = 207.471 us, min = 15.170 us, total = 7.318 ms |
|
NodeInfoGcsService.grpc_server.CheckAlive - 72 total (0 active), Execution time: mean = 521.294 us, total = 37.533 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
ClusterResourceManager.ResetRemoteNodeView - 60 total (1 active), Execution time: mean = 9.533 us, total = 571.975 us, Queueing time: mean = 97.336 us, max = 1.634 ms, min = 26.287 us, total = 5.840 ms |
|
HealthCheck - 58 total (0 active), Execution time: mean = 7.674 us, total = 445.084 us, Queueing time: mean = 101.178 us, max = 204.653 us, min = 20.297 us, total = 5.868 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 36 total (0 active), Execution time: mean = 78.946 us, total = 2.842 ms, Queueing time: mean = 98.087 us, max = 201.135 us, min = 20.386 us, total = 3.531 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 36 total (0 active), Execution time: mean = 581.580 us, total = 20.937 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GCSServer.deadline_timer.debug_state_dump - 18 total (1 active), Execution time: mean = 1.291 ms, total = 23.234 ms, Queueing time: mean = 52.790 us, max = 119.431 us, min = 16.811 us, total = 950.218 us |
|
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us |
|
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us |
|
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms |
|
GCSServer.deadline_timer.debug_state_event_stats_print - 3 total (1 active, 1 running), Execution time: mean = 1.029 ms, total = 3.087 ms, Queueing time: mean = 49.251 us, max = 77.585 us, min = 70.167 us, total = 147.752 us |
|
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us |
|
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us |
|
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us |
|
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns |
|
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us |
|
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us |
|
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us |
|
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us |
|
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us |
|
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 8236 total (1 active) |
|
Queueing time: mean = 44.822 us, max = 1.349 ms, min = 2.131 us, total = 369.154 ms |
|
Execution time: mean = 230.525 us, total = 1.899 s |
|
Event stats: |
|
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 3738 total (0 active), Execution time: mean = 22.212 us, total = 83.029 ms, Queueing time: mean = 96.123 us, max = 1.349 ms, min = 5.172 us, total = 359.306 ms |
|
TaskInfoGcsService.grpc_server.AddTaskEventData - 3738 total (0 active), Execution time: mean = 480.757 us, total = 1.797 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.119 us, total = 18.136 ms, Queueing time: mean = 9.202 us, max = 389.975 us, min = 2.131 us, total = 6.644 ms |
|
GcsTaskManager.GcJobSummary - 37 total (1 active), Execution time: mean = 9.707 us, total = 359.160 us, Queueing time: mean = 86.250 us, max = 606.639 us, min = 32.578 us, total = 3.191 ms |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us |
|
|
|
|
|
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 978 total (25 active) |
|
Queueing time: mean = 21.875 us, max = 2.957 ms, min = 3.295 us, total = 21.394 ms |
|
Execution time: mean = 177.887 ms, total = 173.973 s |
|
Event stats: |
|
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 25.132 us, total = 18.145 ms, Queueing time: mean = 13.500 us, max = 2.957 ms, min = 3.295 us, total = 9.747 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 63 total (24 active), Execution time: mean = 2.760 s, total = 173.909 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 63 total (0 active), Execution time: mean = 34.135 us, total = 2.150 ms, Queueing time: mean = 81.435 us, max = 239.158 us, min = 13.433 us, total = 5.130 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 39 total (0 active), Execution time: mean = 104.532 us, total = 4.077 ms, Queueing time: mean = 82.781 us, max = 175.467 us, min = 17.549 us, total = 3.228 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish - 39 total (0 active), Execution time: mean = 540.745 us, total = 21.089 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:01:18,986 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 728 total (0 active) |
|
Queueing time: mean = 11.416 us, max = 1.158 ms, min = 692.000 ns, total = 8.311 ms |
|
Execution time: mean = 26.389 us, total = 19.211 ms |
|
Event stats: |
|
event_loop_lag_probe - 722 total (0 active), Execution time: mean = 26.340 us, total = 19.018 ms, Queueing time: mean = 11.409 us, max = 1.158 ms, min = 4.729 us, total = 8.238 ms |
|
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us |
|
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us |
|
|
|
|
|
[2025-01-21 06:02:18,986 I 20340 20340] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 1 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 34 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 48 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
RAY_LOG_CHANNEL |
|
- cumulative published messages: 51 |
|
- cumulative published bytes: 41859 |
|
- current buffered bytes: 818 |
|
GCS_NODE_INFO_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 536 |
|
- current buffered bytes: 0 |
|
GCS_JOB_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 283 |
|
- current buffered bytes: 283 |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 3 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 2 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 1 |
|
-Total num of driver tasks: 1 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 3274 total (4 active) |
|
Queueing time: mean = 626.158 us, max = 209.811 ms, min = 740.000 ns, total = 2.050 s |
|
Execution time: mean = 288.581 us, total = 944.813 ms |
|
Event stats: |
|
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 22.636 us, total = 21.776 ms, Queueing time: mean = 15.383 us, max = 6.396 ms, min = 1.699 us, total = 14.798 ms |
|
GcsInMemoryStore.Put - 431 total (0 active), Execution time: mean = 493.935 us, total = 212.886 ms, Queueing time: mean = 3.441 ms, max = 209.465 ms, min = 1.325 us, total = 1.483 s |
|
RayletLoadPulled - 240 total (1 active), Execution time: mean = 236.422 us, total = 56.741 ms, Queueing time: mean = 64.538 us, max = 183.264 us, min = 10.641 us, total = 15.489 ms |
|
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 237 total (0 active), Execution time: mean = 70.073 us, total = 16.607 ms, Queueing time: mean = 88.671 us, max = 183.295 us, min = 15.696 us, total = 21.015 ms |
|
NodeManagerService.grpc_client.GetResourceLoad - 237 total (0 active), Execution time: mean = 1.386 ms, total = 328.371 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 111 total (0 active), Execution time: mean = 32.947 us, total = 3.657 ms, Queueing time: mean = 72.641 us, max = 203.571 us, min = 8.800 us, total = 8.063 ms |
|
InternalKVGcsService.grpc_server.InternalKVPut - 111 total (0 active), Execution time: mean = 455.889 us, total = 50.604 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Get - 108 total (0 active), Execution time: mean = 18.864 us, total = 2.037 ms, Queueing time: mean = 18.132 us, max = 158.434 us, min = 2.067 us, total = 1.958 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 107 total (0 active), Execution time: mean = 38.621 us, total = 4.132 ms, Queueing time: mean = 310.524 us, max = 8.337 ms, min = 11.662 us, total = 33.226 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet - 107 total (0 active), Execution time: mean = 711.148 us, total = 76.093 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 96 total (0 active), Execution time: mean = 28.675 us, total = 2.753 ms, Queueing time: mean = 92.781 us, max = 207.471 us, min = 15.170 us, total = 8.907 ms |
|
NodeInfoGcsService.grpc_server.CheckAlive - 96 total (0 active), Execution time: mean = 494.699 us, total = 47.491 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
ClusterResourceManager.ResetRemoteNodeView - 80 total (1 active), Execution time: mean = 9.078 us, total = 726.257 us, Queueing time: mean = 84.793 us, max = 1.634 ms, min = 13.790 us, total = 6.783 ms |
|
HealthCheck - 78 total (0 active), Execution time: mean = 7.426 us, total = 579.227 us, Queueing time: mean = 90.262 us, max = 204.653 us, min = 16.570 us, total = 7.040 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 48 total (0 active), Execution time: mean = 75.505 us, total = 3.624 ms, Queueing time: mean = 92.063 us, max = 201.135 us, min = 19.197 us, total = 4.419 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 48 total (0 active), Execution time: mean = 550.398 us, total = 26.419 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 35.401 us, total = 1.204 ms, Queueing time: mean = 590.521 us, max = 8.472 ms, min = 11.517 us, total = 20.078 ms |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 938.702 us, total = 31.916 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GCSServer.deadline_timer.debug_state_dump - 24 total (1 active), Execution time: mean = 1.325 ms, total = 31.793 ms, Queueing time: mean = 51.273 us, max = 119.431 us, min = 16.811 us, total = 1.231 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 26.838 us, total = 563.603 us, Queueing time: mean = 126.532 us, max = 257.619 us, min = 30.842 us, total = 2.657 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 491.931 us, total = 10.331 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 8.268 us, total = 49.609 us, Queueing time: mean = 32.229 us, max = 39.816 us, min = 10.594 us, total = 193.373 us |
|
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 199.609 us, total = 998.045 us, Queueing time: mean = 58.098 us, max = 128.737 us, min = 15.591 us, total = 290.490 us |
|
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 685.197 us, total = 3.426 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 117.076 us, total = 468.306 us, Queueing time: mean = 104.964 ms, max = 209.811 ms, min = 27.036 us, total = 419.855 ms |
|
GCSServer.deadline_timer.debug_state_event_stats_print - 4 total (1 active, 1 running), Execution time: mean = 1.158 ms, total = 4.632 ms, Queueing time: mean = 53.479 us, max = 77.585 us, min = 66.164 us, total = 213.916 us |
|
GcsResourceManager::Update - 2 total (0 active), Execution time: mean = 82.721 us, total = 165.442 us, Queueing time: mean = 106.719 us, max = 117.209 us, min = 96.230 us, total = 213.439 us |
|
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.530 us, total = 9.530 us, Queueing time: mean = 5.988 us, max = 5.988 us, min = 5.988 us, total = 5.988 us |
|
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 54.650 us, total = 54.650 us, Queueing time: mean = 15.901 us, max = 15.901 us, min = 15.901 us, total = 15.901 us |
|
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 46.994 us, total = 46.994 us, Queueing time: mean = 94.708 us, max = 94.708 us, min = 94.708 us, total = 94.708 us |
|
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 33.033 us, total = 33.033 us, Queueing time: mean = 105.695 us, max = 105.695 us, min = 105.695 us, total = 105.695 us |
|
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 17.533 us, total = 17.533 us, Queueing time: mean = 94.945 us, max = 94.945 us, min = 94.945 us, total = 94.945 us |
|
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 492.808 us, total = 492.808 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 920.233 us, total = 920.233 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 23.201 us, total = 23.201 us, Queueing time: mean = 89.441 us, max = 89.441 us, min = 89.441 us, total = 89.441 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 395.533 us, total = 395.533 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 431.027 us, total = 431.027 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 975.673 us, total = 975.673 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 513.632 us, total = 513.632 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 19.693 us, total = 19.693 us, Queueing time: mean = 10.630 us, max = 10.630 us, min = 10.630 us, total = 10.630 us |
|
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 209.342 us, total = 209.342 us, Queueing time: mean = 134.661 us, max = 134.661 us, min = 134.661 us, total = 134.661 us |
|
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 479.251 us, total = 479.251 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 130.738 us, total = 130.738 us, Queueing time: mean = 148.481 us, max = 148.481 us, min = 148.481 us, total = 148.481 us |
|
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.717 us, total = 15.717 us, Queueing time: mean = 740.000 ns, max = 740.000 ns, min = 740.000 ns, total = 740.000 ns |
|
|
|
|
|
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 11006 total (1 active) |
|
Queueing time: mean = 41.435 us, max = 1.349 ms, min = 2.131 us, total = 456.032 ms |
|
Execution time: mean = 218.563 us, total = 2.406 s |
|
Event stats: |
|
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 4997 total (0 active), Execution time: mean = 21.462 us, total = 107.245 ms, Queueing time: mean = 88.807 us, max = 1.349 ms, min = 3.632 us, total = 443.771 ms |
|
TaskInfoGcsService.grpc_server.AddTaskEventData - 4997 total (0 active), Execution time: mean = 455.157 us, total = 2.274 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.292 us, total = 23.369 ms, Queueing time: mean = 8.789 us, max = 389.975 us, min = 2.131 us, total = 8.455 ms |
|
GcsTaskManager.GcJobSummary - 49 total (1 active), Execution time: mean = 9.453 us, total = 463.209 us, Queueing time: mean = 77.432 us, max = 606.639 us, min = 29.279 us, total = 3.794 ms |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 7.846 us, total = 7.846 us, Queueing time: mean = 12.503 us, max = 12.503 us, min = 12.503 us, total = 12.503 us |
|
|
|
|
|
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 1266 total (25 active) |
|
Queueing time: mean = 20.072 us, max = 2.957 ms, min = 2.399 us, total = 25.411 ms |
|
Execution time: mean = 184.984 ms, total = 234.190 s |
|
Event stats: |
|
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 24.642 us, total = 23.706 ms, Queueing time: mean = 12.306 us, max = 2.957 ms, min = 2.399 us, total = 11.838 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 75 total (24 active), Execution time: mean = 3.121 s, total = 234.112 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 75 total (0 active), Execution time: mean = 35.100 us, total = 2.632 ms, Queueing time: mean = 82.030 us, max = 239.158 us, min = 13.433 us, total = 6.152 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 51 total (0 active), Execution time: mean = 107.728 us, total = 5.494 ms, Queueing time: mean = 81.036 us, max = 175.467 us, min = 17.549 us, total = 4.133 ms |
|
InternalPubSubGcsService.grpc_server.GcsPublish - 51 total (0 active), Execution time: mean = 545.307 us, total = 27.811 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 41.727 us, total = 1.043 ms, Queueing time: mean = 71.718 us, max = 203.138 us, min = 5.341 us, total = 1.793 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 712.078 us, total = 17.802 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 55.850 us, total = 55.850 us, Queueing time: mean = 1.494 ms, max = 1.494 ms, min = 1.494 ms, total = 1.494 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:02:18,987 I 20340 20340] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 968 total (0 active) |
|
Queueing time: mean = 11.091 us, max = 1.158 ms, min = 692.000 ns, total = 10.736 ms |
|
Execution time: mean = 26.391 us, total = 25.546 ms |
|
Event stats: |
|
event_loop_lag_probe - 962 total (0 active), Execution time: mean = 26.354 us, total = 25.353 ms, Queueing time: mean = 11.084 us, max = 1.158 ms, min = 4.425 us, total = 10.662 ms |
|
- 2 total (0 active), Execution time: mean = 77.838 us, total = 155.677 us, Queueing time: mean = 22.479 us, max = 27.307 us, min = 17.651 us, total = 44.958 us |
|
RaySyncer.BroadcastMessage - 2 total (0 active), Execution time: mean = 18.387 us, total = 36.773 us, Queueing time: mean = 851.500 ns, max = 1.011 us, min = 692.000 ns, total = 1.703 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 558.000 ns, total = 1.116 us, Queueing time: mean = 13.246 us, max = 14.307 us, min = 12.186 us, total = 26.493 us |
|
|
|
|
|
|
