|
[2025-01-21 06:02:59,007 I 22054 22054] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
[2025-01-21 06:02:59,008 I 22054 22054] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service. |
|
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for GCS |
|
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE |
|
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR |
|
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB |
|
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:324: Set ray event level to warning |
|
[2025-01-21 06:02:59,019 I 22054 22054] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:42: Loading job table data. |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:54: Loading node table data. |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:80: Loading actor table data. |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data. |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:66: Loading placement group table data. |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0 |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0 |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0 |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0 |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0 |
|
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: b89f5df64581a09eded9a19cd2dde8d34725a01ff053e2fab1e5f979 |
|
[2025-01-21 06:02:59,021 I 22054 22054] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0 |
|
[2025-01-21 06:02:59,023 I 22054 22054] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 44745. |
|
[2025-01-21 06:02:59,289 I 22054 22054] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 0 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 0 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 0 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 0 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 0 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 0 |
|
-Total num of driver tasks: 0 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 24 total (5 active) |
|
Queueing time: mean = 101.135 ms, max = 268.934 ms, min = 3.420 us, total = 2.427 s |
|
Execution time: mean = 11.215 ms, total = 269.162 ms |
|
Event stats: |
|
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 29.886 ms, total = 268.977 ms, Queueing time: mean = 208.505 ms, max = 268.332 ms, min = 5.511 us, total = 1.877 s |
|
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 10.316 us, total = 51.580 us, Queueing time: mean = 64.072 us, max = 68.608 us, min = 58.664 us, total = 320.360 us |
|
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.466 us, total = 9.864 us, Queueing time: mean = 134.431 ms, max = 268.934 ms, min = 268.789 ms, total = 537.723 ms |
|
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 9.021 us, total = 18.041 us, Queueing time: mean = 6.323 ms, max = 12.430 ms, min = 214.999 us, total = 12.645 ms |
|
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 105.420 us, total = 105.420 us, Queueing time: mean = 3.420 us, max = 3.420 us, min = 3.420 us, total = 3.420 us |
|
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 5 total (1 active) |
|
Queueing time: mean = 599.243 us, max = 2.093 ms, min = 10.377 us, total = 2.996 ms |
|
Execution time: mean = 324.974 us, total = 1.625 ms |
|
Event stats: |
|
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 536.291 us, total = 1.609 ms, Queueing time: mean = 986.666 us, max = 2.093 ms, min = 10.377 us, total = 2.960 ms |
|
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.998 us, total = 15.998 us, Queueing time: mean = 36.218 us, max = 36.218 us, min = 36.218 us, total = 36.218 us |
|
|
|
|
|
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 5 total (1 active) |
|
Queueing time: mean = 1.365 ms, max = 6.582 ms, min = 9.192 us, total = 6.823 ms |
|
Execution time: mean = 42.756 us, total = 213.781 us |
|
Event stats: |
|
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 54.696 us, total = 164.087 us, Queueing time: mean = 2.229 ms, max = 6.582 ms, min = 9.192 us, total = 6.688 ms |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 49.694 us, total = 49.694 us, Queueing time: mean = 134.431 us, max = 134.431 us, min = 134.431 us, total = 134.431 us |
|
|
|
|
|
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 5 total (0 active) |
|
Queueing time: mean = 124.960 us, max = 565.420 us, min = 8.197 us, total = 624.798 us |
|
Execution time: mean = 100.403 us, total = 502.015 us |
|
Event stats: |
|
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 167.005 us, total = 501.016 us, Queueing time: mean = 198.087 us, max = 565.420 us, min = 8.197 us, total = 594.262 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 499.500 ns, total = 999.000 ns, Queueing time: mean = 15.268 us, max = 20.505 us, min = 10.031 us, total = 30.536 us |
|
|
|
|
|
[2025-01-21 06:03:01,451 I 22054 22054] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:03:01,452 I 22054 22054] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:03:01,452 I 22054 22054] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 registered, will try to reschedule all the infeasible placement groups. |
|
[2025-01-21 06:03:01,456 I 22054 22132] (gcs_server) ray_syncer.cc:377: Get connection node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:03:02,357 I 22054 22054] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 21983 |
|
[2025-01-21 06:03:02,357 I 22054 22054] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 21983 |
|
[2025-01-21 06:03:09,026 W 22054 22077] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses |
|
[2025-01-21 06:03:59,290 I 22054 22054] (gcs_server) gcs_server.cc:245: Gcs Debug state: |
|
|
|
GcsNodeManager: |
|
- RegisterNode request count: 1 |
|
- DrainNode request count: 0 |
|
- GetAllNodeInfo request count: 34 |
|
|
|
GcsActorManager: |
|
- RegisterActor request count: 0 |
|
- CreateActor request count: 0 |
|
- GetActorInfo request count: 0 |
|
- GetNamedActorInfo request count: 0 |
|
- GetAllActorInfo request count: 0 |
|
- KillActor request count: 0 |
|
- ListNamedActors request count: 0 |
|
- Registered actors count: 0 |
|
- Destroyed actors count: 0 |
|
- Named actors count: 0 |
|
- Unresolved actors count: 0 |
|
- Pending actors count: 0 |
|
- Created actors count: 0 |
|
- owners_: 0 |
|
- actor_to_register_callbacks_: 0 |
|
- actor_to_restart_callbacks_: 0 |
|
- actor_to_create_callbacks_: 0 |
|
- sorted_destroyed_actor_list_: 0 |
|
|
|
GcsResourceManager: |
|
- GetAllAvailableResources request count: 0 |
|
- GetAllTotalResources request count: 0 |
|
- GetAllResourceUsage request count: 12 |
|
|
|
GcsPlacementGroupManager: |
|
- CreatePlacementGroup request count: 0 |
|
- RemovePlacementGroup request count: 0 |
|
- GetPlacementGroup request count: 0 |
|
- GetAllPlacementGroup request count: 0 |
|
- WaitPlacementGroupUntilReady request count: 0 |
|
- GetNamedPlacementGroup request count: 0 |
|
- Scheduling pending placement group count: 0 |
|
- Registered placement groups count: 0 |
|
- Named placement group count: 0 |
|
- Pending placement groups count: 0 |
|
- Infeasible placement groups count: 0 |
|
|
|
Publisher: |
|
RAY_LOG_CHANNEL |
|
- cumulative published messages: 14 |
|
- cumulative published bytes: 12693 |
|
- current buffered bytes: 818 |
|
GCS_NODE_INFO_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 536 |
|
- current buffered bytes: 0 |
|
GCS_JOB_CHANNEL |
|
- cumulative published messages: 1 |
|
- cumulative published bytes: 283 |
|
- current buffered bytes: 283 |
|
|
|
[runtime env manager] ID to URIs table: |
|
[runtime env manager] URIs reference table: |
|
|
|
GcsTaskManager: |
|
-Total num task events reported: 7 |
|
-Total num status task events dropped: 0 |
|
-Total num profile events dropped: 0 |
|
-Current num of task events stored: 2 |
|
-Total num of actor creation tasks: 0 |
|
-Total num of actor tasks: 0 |
|
-Total num of normal tasks: 1 |
|
-Total num of driver tasks: 1 |
|
|
|
GcsAutoscalerStateManager: |
|
- last_seen_autoscaler_state_version_: 0 |
|
- last_cluster_resource_state_version_: 0 |
|
- pending demands: |
|
|
|
|
|
|
|
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:843: Main service Event stats: |
|
|
|
|
|
Global stats: 1118 total (4 active) |
|
Queueing time: mean = 2.227 ms, max = 268.934 ms, min = 826.000 ns, total = 2.489 s |
|
Execution time: mean = 454.223 us, total = 507.821 ms |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 21.312 us, total = 5.158 ms, Queueing time: mean = 76.311 us, max = 12.430 ms, min = 2.350 us, total = 18.467 ms |
|
GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.892 ms, total = 270.539 ms, Queueing time: mean = 13.152 ms, max = 268.332 ms, min = 2.510 us, total = 1.881 s |
|
GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 17.049 us, total = 1.228 ms, Queueing time: mean = 10.499 us, max = 48.127 us, min = 2.892 us, total = 755.921 us |
|
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 35.831 us, total = 2.544 ms, Queueing time: mean = 435.552 us, max = 26.736 ms, min = 8.273 us, total = 30.924 ms |
|
InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 780.310 us, total = 55.402 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
RayletLoadPulled - 60 total (1 active), Execution time: mean = 220.083 us, total = 13.205 ms, Queueing time: mean = 65.979 us, max = 637.852 us, min = 17.576 us, total = 3.959 ms |
|
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 70.011 us, total = 3.991 ms, Queueing time: mean = 76.458 us, max = 183.664 us, min = 17.148 us, total = 4.358 ms |
|
NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.350 ms, total = 76.975 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 492.123 us, total = 19.193 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 32.854 us, total = 1.281 ms, Queueing time: mean = 77.953 us, max = 173.701 us, min = 10.260 us, total = 3.040 ms |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 240.807 us, total = 8.187 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 22.723 us, total = 772.592 us, Queueing time: mean = 29.774 us, max = 119.797 us, min = 9.270 us, total = 1.012 ms |
|
NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 482.437 us, total = 11.578 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 28.630 us, total = 687.114 us, Queueing time: mean = 82.252 us, max = 158.434 us, min = 16.029 us, total = 1.974 ms |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 22.746 us, total = 477.664 us, Queueing time: mean = 24.887 us, max = 135.149 us, min = 7.083 us, total = 522.634 us |
|
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 221.944 us, total = 4.661 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 8.755 us, total = 175.104 us, Queueing time: mean = 63.548 us, max = 179.759 us, min = 24.657 us, total = 1.271 ms |
|
HealthCheck - 18 total (0 active), Execution time: mean = 7.603 us, total = 136.857 us, Queueing time: mean = 85.115 us, max = 174.008 us, min = 17.322 us, total = 1.532 ms |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 602.206 us, total = 7.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 79.921 us, total = 959.056 us, Queueing time: mean = 84.807 us, max = 179.900 us, min = 16.395 us, total = 1.018 ms |
|
GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.352 ms, total = 8.114 ms, Queueing time: mean = 48.104 us, max = 91.674 us, min = 28.812 us, total = 288.625 us |
|
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.658 us, total = 75.950 us, Queueing time: mean = 54.785 us, max = 68.608 us, min = 8.350 us, total = 328.710 us |
|
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 786.516 us, total = 3.933 ms, Queueing time: mean = 129.209 us, max = 459.564 us, min = 17.179 us, total = 646.047 us |
|
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.280 ms, total = 6.400 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 176.768 us, total = 707.074 us, Queueing time: mean = 134.566 ms, max = 268.934 ms, min = 48.479 us, total = 538.266 ms |
|
GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 91.930 us, total = 275.789 us, Queueing time: mean = 67.706 us, max = 168.564 us, min = 16.595 us, total = 203.117 us |
|
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 48.214 us, total = 48.214 us, Queueing time: mean = 14.807 us, max = 14.807 us, min = 14.807 us, total = 14.807 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 263.477 us, total = 263.477 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 183.559 us, total = 183.559 us, Queueing time: mean = 26.734 us, max = 26.734 us, min = 26.734 us, total = 26.734 us |
|
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 627.222 us, total = 627.222 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 888.621 us, total = 888.621 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 479.038 us, total = 479.038 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 496.086 us, total = 496.086 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 120.835 us, total = 120.835 us, Queueing time: mean = 15.007 us, max = 15.007 us, min = 15.007 us, total = 15.007 us |
|
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 175.328 us, total = 175.328 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.452 us, total = 49.452 us, Queueing time: mean = 29.816 us, max = 29.816 us, min = 29.816 us, total = 29.816 us |
|
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 474.950 us, total = 474.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.070 us, total = 17.070 us, Queueing time: mean = 11.530 us, max = 11.530 us, min = 11.530 us, total = 11.530 us |
|
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.381 us, total = 22.381 us, Queueing time: mean = 14.581 us, max = 14.581 us, min = 14.581 us, total = 14.581 us |
|
GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.883 us, total = 9.883 us, Queueing time: mean = 6.040 us, max = 6.040 us, min = 6.040 us, total = 6.040 us |
|
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.960 us, total = 58.960 us, Queueing time: mean = 29.528 us, max = 29.528 us, min = 29.528 us, total = 29.528 us |
|
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.692 us, total = 15.692 us, Queueing time: mean = 826.000 ns, max = 826.000 ns, min = 826.000 ns, total = 826.000 ns |
|
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 8.767 us, total = 8.767 us, Queueing time: mean = 14.568 us, max = 14.568 us, min = 14.568 us, total = 14.568 us |
|
|
|
|
|
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: task_io_context Event stats: |
|
|
|
|
|
Global stats: 2682 total (1 active) |
|
Queueing time: mean = 41.591 us, max = 2.093 ms, min = 2.955 us, total = 111.548 ms |
|
Execution time: mean = 220.587 us, total = 591.614 ms |
|
Event stats: |
|
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1213 total (0 active), Execution time: mean = 23.943 us, total = 29.043 ms, Queueing time: mean = 86.779 us, max = 252.430 us, min = 7.922 us, total = 105.263 ms |
|
TaskInfoGcsService.grpc_server.AddTaskEventData - 1213 total (0 active), Execution time: mean = 456.514 us, total = 553.751 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 35.873 us, total = 8.681 ms, Queueing time: mean = 22.272 us, max = 2.093 ms, min = 2.955 us, total = 5.390 ms |
|
GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 9.433 us, total = 122.631 us, Queueing time: mean = 66.134 us, max = 107.056 us, min = 41.987 us, total = 859.743 us |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.998 us, total = 15.998 us, Queueing time: mean = 36.218 us, max = 36.218 us, min = 36.218 us, total = 36.218 us |
|
|
|
|
|
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats: |
|
|
|
|
|
Global stats: 398 total (25 active) |
|
Queueing time: mean = 31.627 us, max = 6.582 ms, min = 3.500 us, total = 12.587 ms |
|
Execution time: mean = 134.891 ms, total = 53.687 s |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 23.758 us, total = 5.749 ms, Queueing time: mean = 35.980 us, max = 6.582 ms, min = 3.500 us, total = 8.707 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 38 total (0 active), Execution time: mean = 25.453 us, total = 967.200 us, Queueing time: mean = 48.413 us, max = 200.945 us, min = 8.145 us, total = 1.840 ms |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 38 total (24 active), Execution time: mean = 1.412 s, total = 53.662 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 33.844 us, total = 846.104 us, Queueing time: mean = 23.059 us, max = 148.427 us, min = 10.775 us, total = 576.482 us |
|
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 270.210 us, total = 6.755 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsPublish - 14 total (0 active), Execution time: mean = 596.557 us, total = 8.352 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 14 total (0 active), Execution time: mean = 127.060 us, total = 1.779 ms, Queueing time: mean = 94.980 us, max = 141.203 us, min = 24.866 us, total = 1.330 ms |
|
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 49.694 us, total = 49.694 us, Queueing time: mean = 134.431 us, max = 134.431 us, min = 134.431 us, total = 134.431 us |
|
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s |
|
|
|
|
|
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats: |
|
|
|
|
|
Global stats: 250 total (0 active) |
|
Queueing time: mean = 12.614 us, max = 565.420 us, min = 710.000 ns, total = 3.154 ms |
|
Execution time: mean = 27.755 us, total = 6.939 ms |
|
Event stats: |
|
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 27.350 us, total = 6.619 ms, Queueing time: mean = 12.092 us, max = 565.420 us, min = 3.663 us, total = 2.926 ms |
|
RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 25.461 us, total = 76.383 us, Queueing time: mean = 886.000 ns, max = 1.163 us, min = 710.000 ns, total = 2.658 us |
|
- 3 total (0 active), Execution time: mean = 80.935 us, total = 242.804 us, Queueing time: mean = 64.717 us, max = 140.184 us, min = 22.375 us, total = 194.151 us |
|
RaySyncerRegister - 2 total (0 active), Execution time: mean = 499.500 ns, total = 999.000 ns, Queueing time: mean = 15.268 us, max = 20.505 us, min = 10.031 us, total = 30.536 us |
|
|
|
|
|
[2025-01-21 06:04:08,210 I 22054 22054] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000 |
|
[2025-01-21 06:04:08,259 I 22054 22054] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,364 I 22054 22103] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,364 I 22054 22103] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 |
|
[2025-01-21 06:04:08,373 I 22054 22054] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down... |
|
[2025-01-21 06:04:08,375 I 22054 22054] (gcs_server) gcs_server.cc:267: Stopping GCS server. |
|
[2025-01-21 06:04:08,454 I 22054 22054] (gcs_server) gcs_server.cc:284: GCS server stopped. |
|
[2025-01-21 06:04:08,455 I 22054 22054] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped. |
|
[2025-01-21 06:04:08,491 I 22054 22054] (gcs_server) stats.h:120: Stats module has shutdown. |
|
|
