File size: 11,537 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
[2025-01-15 22:08:43,597 I 7178 7178] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 22:08:43,597 I 7178 7178] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 22:08:43,603 I 7178 7178] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 22:08:43,603 I 7178 7178] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 22:08:43,603 I 7178 7178] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 22:08:43,603 I 7178 7178] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 22:08:43,603 I 7178 7178] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 22:08:43,612 I 7178 7178] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 4c984220e8f72bdb4e49643d6227f3cb0ad024bb90fe4fb064b3f2ac
[2025-01-15 22:08:43,613 I 7178 7178] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 22:08:43,618 I 7178 7178] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 63497.
[2025-01-15 22:08:43,874 I 7178 7178] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-15 22:08:43,874 I 7178 7178] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 25 total (5 active)
Queueing time: mean = 92.274 ms, max = 255.007 ms, min = 1.747 us, total = 2.307 s
Execution time: mean = 10.455 ms, total = 261.372 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 28.336 ms, total = 255.028 ms, Queueing time: mean = 197.804 ms, max = 254.553 ms, min = 1.747 us, total = 1.780 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 7.855 us, total = 39.276 us, Queueing time: mean = 47.410 us, max = 51.832 us, min = 42.892 us, total = 237.048 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.499 us, total = 9.998 us, Queueing time: mean = 127.475 ms, max = 255.007 ms, min = 254.892 ms, total = 509.899 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 7.306 us, total = 14.611 us, Queueing time: mean = 7.907 ms, max = 15.643 ms, min = 172.291 us, total = 15.815 ms
NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 6.268 ms, total = 6.268 ms, Queueing time: mean = 665.799 us, max = 665.799 us, min = 665.799 us, total = 665.799 us
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 11.740 us, total = 11.740 us, Queueing time: mean = 2.624 us, max = 2.624 us, min = 2.624 us, total = 2.624 us
[2025-01-15 22:08:43,875 I 7178 7178] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.059 ms, max = 4.047 ms, min = 9.348 us, total = 5.297 ms
Execution time: mean = 916.167 us, total = 4.581 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.522 ms, total = 4.565 ms, Queueing time: mean = 1.730 ms, max = 4.047 ms, min = 9.348 us, total = 5.190 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.960 us, total = 15.960 us, Queueing time: mean = 106.713 us, max = 106.713 us, min = 106.713 us, total = 106.713 us
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 22:08:43,875 I 7178 7178] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 2.553 ms, max = 10.939 ms, min = 9.096 us, total = 12.765 ms
Execution time: mean = 315.999 us, total = 1.580 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 420.529 us, total = 1.262 ms, Queueing time: mean = 3.770 ms, max = 10.939 ms, min = 9.096 us, total = 11.311 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 318.406 us, total = 318.406 us, Queueing time: mean = 1.455 ms, max = 1.455 ms, min = 1.455 ms, total = 1.455 ms
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 22:08:43,875 I 7178 7178] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 5 total (0 active)
Queueing time: mean = 323.192 us, max = 857.063 us, min = 10.785 us, total = 1.616 ms
Execution time: mean = 682.652 us, total = 3.413 ms
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.137 ms, total = 3.411 ms, Queueing time: mean = 474.088 us, max = 857.063 us, min = 10.785 us, total = 1.422 ms
RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.019 us, total = 2.038 us, Queueing time: mean = 96.847 us, max = 97.004 us, min = 96.690 us, total = 193.694 us
[2025-01-15 22:08:46,158 I 7178 7178] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:46,158 I 7178 7178] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:46,158 I 7178 7178] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 22:08:46,165 I 7178 7261] (gcs_server) ray_syncer.cc:377: Get connection node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,172 I 7178 7178] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 7111
[2025-01-15 22:08:47,173 I 7178 7178] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 7111
[2025-01-15 22:08:47,272 I 7178 7178] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 22:08:47,452 I 7178 7178] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,453 I 7178 7178] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,453 I 7178 7178] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,453 I 7178 7178] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,581 I 7178 7227] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,581 I 7178 7227] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=1ae331df3c166ab7388e30c064dbb2adb1e4aed1b4ea36f75137b9be
[2025-01-15 22:08:47,615 I 7178 7178] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 22:08:47,617 I 7178 7178] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 22:08:47,695 I 7178 7178] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 22:08:47,695 I 7178 7178] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 22:08:47,707 I 7178 7178] (gcs_server) stats.h:120: Stats module has shutdown.
|