Hugging Face's logo

JayKimDevolved
/
tmp

Transformers
Safetensors
Generated from Trainer
trl
sft
Inference Endpoints
Model card Files Community
tmp
File size: 11,722 Bytes 
c011401
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
[2025-01-15 18:19:32,260 I 551177 551177] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:19:32,260 I 551177 551177] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:19:32,267 I 551177 551177] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:19:32,267 I 551177 551177] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:19:32,267 I 551177 551177] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:19:32,267 I 551177 551177] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:19:32,267 I 551177 551177] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:19:32,275 I 551177 551177] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:19:32,276 I 551177 551177] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 15099f15170aef76500fef75e6ba743865a2ddba76d9061bc194746d
[2025-01-15 18:19:32,277 I 551177 551177] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:19:32,280 I 551177 551177] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 53056.
[2025-01-15 18:19:32,530 I 551177 551177] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-15 18:19:32,531 I 551177 551177] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 90.286 ms, max = 249.335 ms, min = 2.055 us, total = 2.257 s
Execution time:  mean = 10.154 ms, total = 253.843 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.708 ms, total = 249.376 ms, Queueing time: mean = 193.358 ms, max = 248.837 ms, min = 2.055 us, total = 1.740 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 12.562 us, total = 62.811 us, Queueing time: mean = 159.036 us, max = 164.243 us, min = 150.848 us, total = 795.180 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 3.364 us, total = 13.454 us, Queueing time: mean = 124.637 ms, max = 249.335 ms, min = 249.212 ms, total = 498.547 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 52.806 us, total = 105.613 us, Queueing time: mean = 8.263 ms, max = 16.323 ms, min = 203.197 us, total = 16.526 ms
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 14.605 us, total = 14.605 us, Queueing time: mean = 3.322 us, max = 3.322 us, min = 3.322 us, total = 3.322 us
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 4.270 ms, total = 4.270 ms, Queueing time: mean = 1.057 ms, max = 1.057 ms, min = 1.057 ms, total = 1.057 ms
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:19:32,531 I 551177 551177] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 1.669 ms, max = 8.192 ms, min = 9.525 us, total = 8.344 ms
Execution time:  mean = 52.160 us, total = 260.802 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 82.454 us, total = 247.362 us, Queueing time: mean = 2.750 ms, max = 8.192 ms, min = 9.525 us, total = 8.249 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 13.440 us, total = 13.440 us, Queueing time: mean = 95.175 us, max = 95.175 us, min = 95.175 us, total = 95.175 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:19:32,531 I 551177 551177] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 234.358 us, max = 585.168 us, min = 11.580 us, total = 1.172 ms
Execution time:  mean = 568.348 us, total = 2.842 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 929.895 us, total = 2.790 ms, Queueing time: mean = 354.945 us, max = 585.168 us, min = 11.580 us, total = 1.065 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 52.057 us, total = 52.057 us, Queueing time: mean = 106.955 us, max = 106.955 us, min = 106.955 us, total = 106.955 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:19:32,531 I 551177 551177] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 5 total (0 active)
Queueing time: mean = 1.805 ms, max = 8.824 ms, min = 8.166 us, total = 9.026 ms
Execution time:  mean = 47.289 us, total = 236.446 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 78.123 us, total = 234.368 us, Queueing time: mean = 2.954 ms, max = 8.824 ms, min = 8.166 us, total = 8.862 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.039 us, total = 2.078 us, Queueing time: mean = 82.151 us, max = 83.243 us, min = 81.060 us, total = 164.303 us


[2025-01-15 18:19:33,779 I 551177 551177] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:33,780 I 551177 551177] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:33,780 I 551177 551177] (gcs_server) gcs_placement_group_manager.cc:819: A new node: d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8 registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:19:33,786 I 551177 551253] (gcs_server) ray_syncer.cc:377: Get connection node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:34,783 I 551177 551177] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 551110
[2025-01-15 18:19:34,784 I 551177 551177] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 551110
[2025-01-15 18:19:35,001 I 551177 551177] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:19:35,127 I 551177 551177] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,127 I 551177 551177] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,127 I 551177 551177] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,127 I 551177 551177] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,295 I 551177 551226] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,296 I 551177 551226] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=d12be26d548ec0ad5a64e9aba80ca2c1bb2644c497abb7f7489e6da8
[2025-01-15 18:19:35,340 I 551177 551177] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:19:35,342 I 551177 551177] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:19:35,412 I 551177 551177] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:19:35,412 I 551177 551177] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:19:35,470 I 551177 551177] (gcs_server) stats.h:120: Stats module has shutdown.