File size: 28,101 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303 304 305 306 307 308 309 310 311 312 313 314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329 330 331 332 333 334 335 336 337 |
[2025-01-21 06:02:59,007 I 22054 22054] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-21 06:02:59,008 I 22054 22054] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-21 06:02:59,013 I 22054 22054] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-21 06:02:59,019 I 22054 22054] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-21 06:02:59,020 I 22054 22054] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: b89f5df64581a09eded9a19cd2dde8d34725a01ff053e2fab1e5f979
[2025-01-21 06:02:59,021 I 22054 22054] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-21 06:02:59,023 I 22054 22054] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 44745.
[2025-01-21 06:02:59,289 I 22054 22054] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 24 total (5 active)
Queueing time: mean = 101.135 ms, max = 268.934 ms, min = 3.420 us, total = 2.427 s
Execution time: mean = 11.215 ms, total = 269.162 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 29.886 ms, total = 268.977 ms, Queueing time: mean = 208.505 ms, max = 268.332 ms, min = 5.511 us, total = 1.877 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 10.316 us, total = 51.580 us, Queueing time: mean = 64.072 us, max = 68.608 us, min = 58.664 us, total = 320.360 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.466 us, total = 9.864 us, Queueing time: mean = 134.431 ms, max = 268.934 ms, min = 268.789 ms, total = 537.723 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 9.021 us, total = 18.041 us, Queueing time: mean = 6.323 ms, max = 12.430 ms, min = 214.999 us, total = 12.645 ms
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 105.420 us, total = 105.420 us, Queueing time: mean = 3.420 us, max = 3.420 us, min = 3.420 us, total = 3.420 us
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 599.243 us, max = 2.093 ms, min = 10.377 us, total = 2.996 ms
Execution time: mean = 324.974 us, total = 1.625 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 536.291 us, total = 1.609 ms, Queueing time: mean = 986.666 us, max = 2.093 ms, min = 10.377 us, total = 2.960 ms
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.998 us, total = 15.998 us, Queueing time: mean = 36.218 us, max = 36.218 us, min = 36.218 us, total = 36.218 us
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.365 ms, max = 6.582 ms, min = 9.192 us, total = 6.823 ms
Execution time: mean = 42.756 us, total = 213.781 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 54.696 us, total = 164.087 us, Queueing time: mean = 2.229 ms, max = 6.582 ms, min = 9.192 us, total = 6.688 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 49.694 us, total = 49.694 us, Queueing time: mean = 134.431 us, max = 134.431 us, min = 134.431 us, total = 134.431 us
[2025-01-21 06:02:59,290 I 22054 22054] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 5 total (0 active)
Queueing time: mean = 124.960 us, max = 565.420 us, min = 8.197 us, total = 624.798 us
Execution time: mean = 100.403 us, total = 502.015 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 167.005 us, total = 501.016 us, Queueing time: mean = 198.087 us, max = 565.420 us, min = 8.197 us, total = 594.262 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 499.500 ns, total = 999.000 ns, Queueing time: mean = 15.268 us, max = 20.505 us, min = 10.031 us, total = 30.536 us
[2025-01-21 06:03:01,451 I 22054 22054] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:03:01,452 I 22054 22054] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:03:01,452 I 22054 22054] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877 registered, will try to reschedule all the infeasible placement groups.
[2025-01-21 06:03:01,456 I 22054 22132] (gcs_server) ray_syncer.cc:377: Get connection node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:03:02,357 I 22054 22054] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 21983
[2025-01-21 06:03:02,357 I 22054 22054] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 21983
[2025-01-21 06:03:09,026 W 22054 22077] (gcs_server) metric_exporter.cc:105: [1] Export metrics to agent failed: RpcError: RPC Error message: failed to connect to all addresses; last error: UNKNOWN: ipv4:127.0.0.1:50564: Failed to connect to remote host: Connection refused; RPC Error details: . This won't affect Ray, but you can lose metrics from the cluster.
[2025-01-21 06:03:59,290 I 22054 22054] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 1
- DrainNode request count: 0
- GetAllNodeInfo request count: 34
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 12
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
RAY_LOG_CHANNEL
- cumulative published messages: 14
- cumulative published bytes: 12693
- current buffered bytes: 818
GCS_NODE_INFO_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 536
- current buffered bytes: 0
GCS_JOB_CHANNEL
- cumulative published messages: 1
- cumulative published bytes: 283
- current buffered bytes: 283
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 7
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 2
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 1
-Total num of driver tasks: 1
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 1118 total (4 active)
Queueing time: mean = 2.227 ms, max = 268.934 ms, min = 826.000 ns, total = 2.489 s
Execution time: mean = 454.223 us, total = 507.821 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 21.312 us, total = 5.158 ms, Queueing time: mean = 76.311 us, max = 12.430 ms, min = 2.350 us, total = 18.467 ms
GcsInMemoryStore.Put - 143 total (0 active), Execution time: mean = 1.892 ms, total = 270.539 ms, Queueing time: mean = 13.152 ms, max = 268.332 ms, min = 2.510 us, total = 1.881 s
GcsInMemoryStore.Get - 72 total (0 active), Execution time: mean = 17.049 us, total = 1.228 ms, Queueing time: mean = 10.499 us, max = 48.127 us, min = 2.892 us, total = 755.921 us
InternalKVGcsService.grpc_server.InternalKVGet.HandleRequestImpl - 71 total (0 active), Execution time: mean = 35.831 us, total = 2.544 ms, Queueing time: mean = 435.552 us, max = 26.736 ms, min = 8.273 us, total = 30.924 ms
InternalKVGcsService.grpc_server.InternalKVGet - 71 total (0 active), Execution time: mean = 780.310 us, total = 55.402 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
RayletLoadPulled - 60 total (1 active), Execution time: mean = 220.083 us, total = 13.205 ms, Queueing time: mean = 65.979 us, max = 637.852 us, min = 17.576 us, total = 3.959 ms
NodeManagerService.grpc_client.GetResourceLoad.OnReplyReceived - 57 total (0 active), Execution time: mean = 70.011 us, total = 3.991 ms, Queueing time: mean = 76.458 us, max = 183.664 us, min = 17.148 us, total = 4.358 ms
NodeManagerService.grpc_client.GetResourceLoad - 57 total (0 active), Execution time: mean = 1.350 ms, total = 76.975 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut - 39 total (0 active), Execution time: mean = 492.123 us, total = 19.193 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.InternalKVPut.HandleRequestImpl - 39 total (0 active), Execution time: mean = 32.854 us, total = 1.281 ms, Queueing time: mean = 77.953 us, max = 173.701 us, min = 10.260 us, total = 3.040 ms
NodeInfoGcsService.grpc_server.GetAllNodeInfo - 34 total (0 active), Execution time: mean = 240.807 us, total = 8.187 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetAllNodeInfo.HandleRequestImpl - 34 total (0 active), Execution time: mean = 22.723 us, total = 772.592 us, Queueing time: mean = 29.774 us, max = 119.797 us, min = 9.270 us, total = 1.012 ms
NodeInfoGcsService.grpc_server.CheckAlive - 24 total (0 active), Execution time: mean = 482.437 us, total = 11.578 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.CheckAlive.HandleRequestImpl - 24 total (0 active), Execution time: mean = 28.630 us, total = 687.114 us, Queueing time: mean = 82.252 us, max = 158.434 us, min = 16.029 us, total = 1.974 ms
WorkerInfoGcsService.grpc_server.AddWorkerInfo.HandleRequestImpl - 21 total (0 active), Execution time: mean = 22.746 us, total = 477.664 us, Queueing time: mean = 24.887 us, max = 135.149 us, min = 7.083 us, total = 522.634 us
WorkerInfoGcsService.grpc_server.AddWorkerInfo - 21 total (0 active), Execution time: mean = 221.944 us, total = 4.661 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 20 total (1 active), Execution time: mean = 8.755 us, total = 175.104 us, Queueing time: mean = 63.548 us, max = 179.759 us, min = 24.657 us, total = 1.271 ms
HealthCheck - 18 total (0 active), Execution time: mean = 7.603 us, total = 136.857 us, Queueing time: mean = 85.115 us, max = 174.008 us, min = 17.322 us, total = 1.532 ms
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage - 12 total (0 active), Execution time: mean = 602.206 us, total = 7.226 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeResourceInfoGcsService.grpc_server.GetAllResourceUsage.HandleRequestImpl - 12 total (0 active), Execution time: mean = 79.921 us, total = 959.056 us, Queueing time: mean = 84.807 us, max = 179.900 us, min = 16.395 us, total = 1.018 ms
GCSServer.deadline_timer.debug_state_dump - 6 total (1 active), Execution time: mean = 1.352 ms, total = 8.114 ms, Queueing time: mean = 48.104 us, max = 91.674 us, min = 28.812 us, total = 288.625 us
GcsInMemoryStore.GetAll - 6 total (0 active), Execution time: mean = 12.658 us, total = 75.950 us, Queueing time: mean = 54.785 us, max = 68.608 us, min = 8.350 us, total = 328.710 us
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 5 total (0 active), Execution time: mean = 786.516 us, total = 3.933 ms, Queueing time: mean = 129.209 us, max = 459.564 us, min = 17.179 us, total = 646.047 us
NodeInfoGcsService.grpc_server.GetClusterId - 5 total (0 active), Execution time: mean = 1.280 ms, total = 6.400 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 4 total (0 active), Execution time: mean = 176.768 us, total = 707.074 us, Queueing time: mean = 134.566 ms, max = 268.934 ms, min = 48.479 us, total = 538.266 ms
GcsResourceManager::Update - 3 total (0 active), Execution time: mean = 91.930 us, total = 275.789 us, Queueing time: mean = 67.706 us, max = 168.564 us, min = 16.595 us, total = 203.117 us
InternalKVGcsService.grpc_server.InternalKVDel.HandleRequestImpl - 1 total (0 active), Execution time: mean = 48.214 us, total = 48.214 us, Queueing time: mean = 14.807 us, max = 14.807 us, min = 14.807 us, total = 14.807 us
JobInfoGcsService.grpc_server.GetAllJobInfo - 1 total (0 active), Execution time: mean = 263.477 us, total = 263.477 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode.HandleRequestImpl - 1 total (0 active), Execution time: mean = 183.559 us, total = 183.559 us, Queueing time: mean = 26.734 us, max = 26.734 us, min = 26.734 us, total = 26.734 us
InternalKVGcsService.grpc_server.InternalKVExists - 1 total (0 active), Execution time: mean = 627.222 us, total = 627.222 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.RegisterNode - 1 total (0 active), Execution time: mean = 888.621 us, total = 888.621 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob - 1 total (0 active), Execution time: mean = 479.038 us, total = 479.038 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig - 1 total (0 active), Execution time: mean = 496.086 us, total = 496.086 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
JobInfoGcsService.grpc_server.AddJob.HandleRequestImpl - 1 total (0 active), Execution time: mean = 120.835 us, total = 120.835 us, Queueing time: mean = 15.007 us, max = 15.007 us, min = 15.007 us, total = 15.007 us
JobInfoGcsService.grpc_server.GetNextJobID - 1 total (0 active), Execution time: mean = 175.328 us, total = 175.328 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalKVGcsService.grpc_server.GetInternalConfig.HandleRequestImpl - 1 total (0 active), Execution time: mean = 49.452 us, total = 49.452 us, Queueing time: mean = 29.816 us, max = 29.816 us, min = 29.816 us, total = 29.816 us
InternalKVGcsService.grpc_server.InternalKVDel - 1 total (0 active), Execution time: mean = 474.950 us, total = 474.950 us, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Exists - 1 total (0 active), Execution time: mean = 17.070 us, total = 17.070 us, Queueing time: mean = 11.530 us, max = 11.530 us, min = 11.530 us, total = 11.530 us
JobInfoGcsService.grpc_server.GetAllJobInfo.HandleRequestImpl - 1 total (0 active), Execution time: mean = 22.381 us, total = 22.381 us, Queueing time: mean = 14.581 us, max = 14.581 us, min = 14.581 us, total = 14.581 us
GCSServer.deadline_timer.debug_state_event_stats_print - 1 total (1 active, 1 running), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Delete - 1 total (0 active), Execution time: mean = 9.883 us, total = 9.883 us, Queueing time: mean = 6.040 us, max = 6.040 us, min = 6.040 us, total = 6.040 us
InternalKVGcsService.grpc_server.InternalKVExists.HandleRequestImpl - 1 total (0 active), Execution time: mean = 58.960 us, total = 58.960 us, Queueing time: mean = 29.528 us, max = 29.528 us, min = 29.528 us, total = 29.528 us
GcsHealthCheckManager::AddNode - 1 total (0 active), Execution time: mean = 15.692 us, total = 15.692 us, Queueing time: mean = 826.000 ns, max = 826.000 ns, min = 826.000 ns, total = 826.000 ns
JobInfoGcsService.grpc_server.GetNextJobID.HandleRequestImpl - 1 total (0 active), Execution time: mean = 8.767 us, total = 8.767 us, Queueing time: mean = 14.568 us, max = 14.568 us, min = 14.568 us, total = 14.568 us
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 2682 total (1 active)
Queueing time: mean = 41.591 us, max = 2.093 ms, min = 2.955 us, total = 111.548 ms
Execution time: mean = 220.587 us, total = 591.614 ms
Event stats:
TaskInfoGcsService.grpc_server.AddTaskEventData.HandleRequestImpl - 1213 total (0 active), Execution time: mean = 23.943 us, total = 29.043 ms, Queueing time: mean = 86.779 us, max = 252.430 us, min = 7.922 us, total = 105.263 ms
TaskInfoGcsService.grpc_server.AddTaskEventData - 1213 total (0 active), Execution time: mean = 456.514 us, total = 553.751 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 35.873 us, total = 8.681 ms, Queueing time: mean = 22.272 us, max = 2.093 ms, min = 2.955 us, total = 5.390 ms
GcsTaskManager.GcJobSummary - 13 total (1 active), Execution time: mean = 9.433 us, total = 122.631 us, Queueing time: mean = 66.134 us, max = 107.056 us, min = 41.987 us, total = 859.743 us
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.998 us, total = 15.998 us, Queueing time: mean = 36.218 us, max = 36.218 us, min = 36.218 us, total = 36.218 us
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 398 total (25 active)
Queueing time: mean = 31.627 us, max = 6.582 ms, min = 3.500 us, total = 12.587 ms
Execution time: mean = 134.891 ms, total = 53.687 s
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 23.758 us, total = 5.749 ms, Queueing time: mean = 35.980 us, max = 6.582 ms, min = 3.500 us, total = 8.707 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll.HandleRequestImpl - 38 total (0 active), Execution time: mean = 25.453 us, total = 967.200 us, Queueing time: mean = 48.413 us, max = 200.945 us, min = 8.145 us, total = 1.840 ms
InternalPubSubGcsService.grpc_server.GcsSubscriberPoll - 38 total (24 active), Execution time: mean = 1.412 s, total = 53.662 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch.HandleRequestImpl - 25 total (0 active), Execution time: mean = 33.844 us, total = 846.104 us, Queueing time: mean = 23.059 us, max = 148.427 us, min = 10.775 us, total = 576.482 us
InternalPubSubGcsService.grpc_server.GcsSubscriberCommandBatch - 25 total (0 active), Execution time: mean = 270.210 us, total = 6.755 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish - 14 total (0 active), Execution time: mean = 596.557 us, total = 8.352 ms, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
InternalPubSubGcsService.grpc_server.GcsPublish.HandleRequestImpl - 14 total (0 active), Execution time: mean = 127.060 us, total = 1.779 ms, Queueing time: mean = 94.980 us, max = 141.203 us, min = 24.866 us, total = 1.330 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 49.694 us, total = 49.694 us, Queueing time: mean = 134.431 us, max = 134.431 us, min = 134.431 us, total = 134.431 us
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-21 06:03:59,291 I 22054 22054] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 250 total (0 active)
Queueing time: mean = 12.614 us, max = 565.420 us, min = 710.000 ns, total = 3.154 ms
Execution time: mean = 27.755 us, total = 6.939 ms
Event stats:
event_loop_lag_probe - 242 total (0 active), Execution time: mean = 27.350 us, total = 6.619 ms, Queueing time: mean = 12.092 us, max = 565.420 us, min = 3.663 us, total = 2.926 ms
RaySyncer.BroadcastMessage - 3 total (0 active), Execution time: mean = 25.461 us, total = 76.383 us, Queueing time: mean = 886.000 ns, max = 1.163 us, min = 710.000 ns, total = 2.658 us
- 3 total (0 active), Execution time: mean = 80.935 us, total = 242.804 us, Queueing time: mean = 64.717 us, max = 140.184 us, min = 22.375 us, total = 194.151 us
RaySyncerRegister - 2 total (0 active), Execution time: mean = 499.500 ns, total = 999.000 ns, Queueing time: mean = 15.268 us, max = 20.505 us, min = 10.031 us, total = 30.536 us
[2025-01-21 06:04:08,210 I 22054 22054] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-21 06:04:08,259 I 22054 22054] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,260 I 22054 22054] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,364 I 22054 22103] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,364 I 22054 22103] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=959d61ca307c5c3e7967dc4f62c340eb34c700e436feb4955e4f5877
[2025-01-21 06:04:08,373 I 22054 22054] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-21 06:04:08,375 I 22054 22054] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-21 06:04:08,454 I 22054 22054] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-21 06:04:08,455 I 22054 22054] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-21 06:04:08,491 I 22054 22054] (gcs_server) stats.h:120: Stats module has shutdown.
|