Hugging Face's logo

JayKimDevolved
/
tmp

Transformers
Safetensors
Generated from Trainer
trl
sft
Inference Endpoints
Model card Files Community
tmp
File size: 11,741 Bytes 
c011401
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
[2025-01-15 18:16:58,238 I 529259 529259] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:16:58,238 I 529259 529259] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:16:58,242 I 529259 529259] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:16:58,242 I 529259 529259] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:16:58,242 I 529259 529259] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:16:58,242 I 529259 529259] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:16:58,243 I 529259 529259] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:16:58,244 I 529259 529259] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:16:58,246 I 529259 529259] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 94ab4d7fe9d91530af69898943173647fbbe0b18eb7fedd34584a99d
[2025-01-15 18:16:58,247 I 529259 529259] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:16:58,249 I 529259 529259] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 50517.
[2025-01-15 18:16:58,463 I 529259 529259] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-15 18:16:58,463 I 529259 529259] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 77.292 ms, max = 214.469 ms, min = 4.900 us, total = 1.932 s
Execution time:  mean = 8.661 ms, total = 216.516 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 23.836 ms, total = 214.523 ms, Queueing time: mean = 165.969 ms, max = 213.660 ms, min = 4.900 us, total = 1.494 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 21.136 us, total = 105.681 us, Queueing time: mean = 166.095 us, max = 191.650 us, min = 135.006 us, total = 830.477 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.858 us, total = 11.432 us, Queueing time: mean = 107.163 ms, max = 214.469 ms, min = 214.183 ms, total = 428.652 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 22.855 us, total = 45.711 us, Queueing time: mean = 4.295 ms, max = 8.147 ms, min = 444.156 us, total = 8.591 ms
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 1.798 ms, total = 1.798 ms, Queueing time: mean = 498.854 us, max = 498.854 us, min = 498.854 us, total = 498.854 us
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 32.123 us, total = 32.123 us, Queueing time: mean = 7.037 us, max = 7.037 us, min = 7.037 us, total = 7.037 us
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:16:58,463 I 529259 529259] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 4 total (1 active)
Queueing time: mean = 98.581 us, max = 296.418 us, min = 13.978 us, total = 394.324 us
Execution time:  mean = 174.935 us, total = 699.741 us
Event stats:
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 346.446 us, total = 692.892 us, Queueing time: mean = 190.173 us, max = 296.418 us, min = 83.928 us, total = 380.346 us
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 6.849 us, total = 6.849 us, Queueing time: mean = 13.978 us, max = 13.978 us, min = 13.978 us, total = 13.978 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:16:58,464 I 529259 529259] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 4 total (1 active)
Queueing time: mean = 531.944 us, max = 2.076 ms, min = 23.426 us, total = 2.128 ms
Execution time:  mean = 56.708 us, total = 226.833 us
Event stats:
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 107.549 us, total = 215.099 us, Queueing time: mean = 1.052 ms, max = 2.076 ms, min = 28.083 us, total = 2.104 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 11.734 us, total = 11.734 us, Queueing time: mean = 23.426 us, max = 23.426 us, min = 23.426 us, total = 23.426 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:16:58,464 I 529259 529259] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 4 total (0 active)
Queueing time: mean = 457.036 us, max = 1.761 ms, min = 15.164 us, total = 1.828 ms
Execution time:  mean = 51.867 us, total = 207.468 us
Event stats:
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 102.953 us, total = 205.905 us, Queueing time: mean = 895.293 us, max = 1.761 ms, min = 29.652 us, total = 1.791 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 781.500 ns, total = 1.563 us, Queueing time: mean = 18.779 us, max = 22.394 us, min = 15.164 us, total = 37.558 us


[2025-01-15 18:17:00,568 I 529259 529259] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:00,568 I 529259 529259] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:00,568 I 529259 529259] (gcs_server) gcs_placement_group_manager.cc:819: A new node: 938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:17:00,572 I 529259 529344] (gcs_server) ray_syncer.cc:377: Get connection node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:01,408 I 529259 529259] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 529191
[2025-01-15 18:17:01,408 I 529259 529259] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 529191
[2025-01-15 18:17:02,562 I 529259 529259] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:17:02,751 I 529259 529259] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:02,751 I 529259 529259] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:02,751 I 529259 529259] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:02,752 I 529259 529259] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:02,981 I 529259 529308] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: 938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:02,981 I 529259 529308] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=938e199b0f7ae3836ea3f46b91680af3ce13e348ac9259343f05c3fa
[2025-01-15 18:17:03,014 I 529259 529259] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:17:03,015 I 529259 529259] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:17:03,084 I 529259 529259] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:17:03,084 I 529259 529259] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:17:03,147 I 529259 529259] (gcs_server) stats.h:120: Stats module has shutdown.