File size: 11,729 Bytes
c011401 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 |
[2025-01-15 18:19:48,289 I 554383 554383] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:19:48,290 I 554383 554383] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:19:48,296 I 554383 554383] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:19:48,297 I 554383 554383] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:19:48,297 I 554383 554383] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:19:48,297 I 554383 554383] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:19:48,297 I 554383 554383] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:19:48,301 I 554383 554383] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:19:48,303 I 554383 554383] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: f4cad5975a1e4632f1384dc63a0e9028ae53e2596acd8b5b5d3fd8cb
[2025-01-15 18:19:48,304 I 554383 554383] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:19:48,308 I 554383 554383] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 60056.
[2025-01-15 18:19:48,577 I 554383 554383] (gcs_server) gcs_server.cc:245: Gcs Debug state:
GcsNodeManager:
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0
GcsActorManager:
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0
GcsResourceManager:
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0
GcsPlacementGroupManager:
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0
Publisher:
[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:
GcsTaskManager:
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0
GcsAutoscalerStateManager:
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:
[2025-01-15 18:19:48,577 I 554383 554383] (gcs_server) gcs_server.cc:843: Main service Event stats:
Global stats: 25 total (5 active)
Queueing time: mean = 96.732 ms, max = 267.708 ms, min = 3.191 us, total = 2.418 s
Execution time: mean = 10.930 ms, total = 273.254 ms
Event stats:
GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 29.748 ms, total = 267.736 ms, Queueing time: mean = 207.388 ms, max = 267.018 ms, min = 3.191 us, total = 1.866 s
GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 13.107 us, total = 65.535 us, Queueing time: mean = 81.633 us, max = 90.353 us, min = 73.670 us, total = 408.163 us
PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.323 us, total = 9.292 us, Queueing time: mean = 133.810 ms, max = 267.708 ms, min = 267.531 ms, total = 535.238 ms
event_loop_lag_probe - 2 total (0 active), Execution time: mean = 11.618 us, total = 23.237 us, Queueing time: mean = 6.905 ms, max = 13.516 ms, min = 293.805 us, total = 13.810 ms
NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 5.400 ms, total = 5.400 ms, Queueing time: mean = 2.342 ms, max = 2.342 ms, min = 2.342 ms, total = 2.342 ms
ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 19.729 us, total = 19.729 us, Queueing time: mean = 4.770 us, max = 4.770 us, min = 4.770 us, total = 4.770 us
RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 18:19:48,577 I 554383 554383] (gcs_server) gcs_server.cc:847: task_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 157.720 us, max = 631.896 us, min = 10.994 us, total = 788.602 us
Execution time: mean = 170.691 us, total = 853.454 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 279.452 us, total = 838.357 us, Queueing time: mean = 232.677 us, max = 631.896 us, min = 10.994 us, total = 698.032 us
GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.097 us, total = 15.097 us, Queueing time: mean = 90.570 us, max = 90.570 us, min = 90.570 us, total = 90.570 us
[2025-01-15 18:19:48,577 I 554383 554383] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:
Global stats: 5 total (1 active)
Queueing time: mean = 1.132 ms, max = 5.492 ms, min = 4.490 us, total = 5.658 ms
Execution time: mean = 51.183 us, total = 255.916 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 80.010 us, total = 240.029 us, Queueing time: mean = 1.846 ms, max = 5.492 ms, min = 4.490 us, total = 5.539 ms
PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.887 us, total = 15.887 us, Queueing time: mean = 118.888 us, max = 118.888 us, min = 118.888 us, total = 118.888 us
Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
[2025-01-15 18:19:48,577 I 554383 554383] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:
Global stats: 5 total (0 active)
Queueing time: mean = 1.042 ms, max = 4.921 ms, min = 4.529 us, total = 5.210 ms
Execution time: mean = 86.955 us, total = 434.774 us
Event stats:
event_loop_lag_probe - 3 total (0 active), Execution time: mean = 144.197 us, total = 432.591 us, Queueing time: mean = 1.655 ms, max = 4.921 ms, min = 4.529 us, total = 4.964 ms
RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.091 us, total = 2.183 us, Queueing time: mean = 122.845 us, max = 124.644 us, min = 121.046 us, total = 245.690 us
[2025-01-15 18:19:50,882 I 554383 554383] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:50,882 I 554383 554383] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:50,882 I 554383 554383] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903 registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:19:50,889 I 554383 554459] (gcs_server) ray_syncer.cc:377: Get connection node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:51,833 I 554383 554383] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 554315
[2025-01-15 18:19:51,834 I 554383 554383] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 554315
[2025-01-15 18:19:53,193 I 554383 554383] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:19:53,257 I 554383 554383] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,257 I 554383 554383] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,257 I 554383 554383] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,257 I 554383 554383] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,399 I 554383 554432] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,400 I 554383 554432] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=21e69873df04ab7aaf4e0c63e69484fc8620911e7569368a47027903
[2025-01-15 18:19:53,420 I 554383 554383] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:19:53,422 I 554383 554383] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:19:53,514 I 554383 554383] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:19:53,514 I 554383 554383] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:19:53,602 I 554383 554383] (gcs_server) stats.h:120: Stats module has shutdown.
|