File size: 11,289 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 |
[2025-01-15 18:16:41,445 I 526049 526049] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:16:41,445 I 526049 526049] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:16:41,450 I 526049 526049] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:16:41,450 I 526049 526049] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:16:41,450 I 526049 526049] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:16:41,450 I 526049 526049] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:16:41,450 I 526049 526049] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:16:41,460 I 526049 526049] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:16:41,462 I 526049 526049] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 6265233987e49e8b266cf57dfc40be1f22be630b00aefc18e382ab4e
[2025-01-15 18:16:41,463 I 526049 526049] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:16:41,467 I 526049 526049] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 60151.
[2025-01-15 18:16:41,711 I 526049 526049] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-15 18:16:41,711 I 526049 526049] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 23 total (4 active)
Queueing time: mean = 97.627 ms, max = 248.229 ms, min = 2.858 us, total = 2.245 s
Execution time: mean = 10.800 ms, total = 248.390 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.587 ms, total = 248.280 ms, Queueing time: mean = 192.365 ms, max = 247.633 ms, min = 2.858 us, total = 1.731 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 11.045 us, total = 55.223 us, Queueing time: mean = 73.216 us, max = 79.817 us, min = 66.850 us, total = 366.082 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 3.454 us, total = 13.818 us, Queueing time: mean = 124.083 ms, max = 248.229 ms, min = 248.102 ms, total = 496.332 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 11.545 us, total = 23.090 us, Queueing time: mean = 8.717 ms, max = 17.180 ms, min = 253.290 us, total = 17.433 ms
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 17.374 us, total = 17.374 us, Queueing time: mean = 3.992 us, max = 3.992 us, min = 3.992 us, total = 3.992 us
[2025-01-15 18:16:41,711 I 526049 526049] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.897 ms, max = 9.019 ms, min = 9.260 us, total = 9.484 ms
Execution time: mean = 371.283 us, total = 1.856 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 614.464 us, total = 1.843 ms, Queueing time: mean = 3.127 ms, max = 9.019 ms, min = 9.260 us, total = 9.381 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 13.025 us, total = 13.025 us, Queueing time: mean = 102.999 us, max = 102.999 us, min = 102.999 us, total = 102.999 us
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 18:16:41,712 I 526049 526049] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.386 ms, max = 5.773 ms, min = 9.010 us, total = 6.932 ms
Execution time: mean = 682.827 us, total = 3.414 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.133 ms, total = 3.400 ms, Queueing time: mean = 2.280 ms, max = 5.773 ms, min = 9.010 us, total = 6.839 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 13.847 us, total = 13.847 us, Queueing time: mean = 92.909 us, max = 92.909 us, min = 92.909 us, total = 92.909 us
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 18:16:41,712 I 526049 526049] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 5 total (0 active)
Queueing time: mean = 838.360 us, max = 3.912 ms, min = 11.226 us, total = 4.192 ms
Execution time: mean = 232.659 us, total = 1.163 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 387.056 us, total = 1.161 ms, Queueing time: mean = 1.318 ms, max = 3.912 ms, min = 11.226 us, total = 3.953 ms
RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.063 us, total = 2.126 us, Queueing time: mean = 119.228 us, max = 121.300 us, min = 117.156 us, total = 238.456 us
[2025-01-15 18:16:43,975 I 526049 526049] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:43,975 I 526049 526049] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:43,975 I 526049 526049] (gcs_server) gcs_placement_group_manager.cc:819: A new node: b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395 registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:16:43,982 I 526049 526135] (gcs_server) ray_syncer.cc:377: Get connection node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:44,943 I 526049 526049] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 525982
[2025-01-15 18:16:44,943 I 526049 526049] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 525982
[2025-01-15 18:16:45,216 I 526049 526049] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:16:45,352 I 526049 526049] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,353 I 526049 526049] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,353 I 526049 526049] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,353 I 526049 526049] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,598 I 526049 526098] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,599 I 526049 526098] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=b31a3685608871f06b74f449b58e09fa860f616ae957b22fb75ad395
[2025-01-15 18:16:45,616 I 526049 526049] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:16:45,618 I 526049 526049] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:16:45,711 I 526049 526049] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:16:45,712 I 526049 526049] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:16:45,755 I 526049 526049] (gcs_server) stats.h:120: Stats module has shutdown.
|