Hugging Face's logo

JayKimDevolved
/
tmp

Transformers
Safetensors
Generated from Trainer
trl
sft
Inference Endpoints
Model card Files Community
tmp
File size: 11,734 Bytes 
c011401
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
[2025-01-15 18:18:19,905 I 539544 539544] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:18:19,905 I 539544 539544] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:18:19,910 I 539544 539544] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:18:19,910 I 539544 539544] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:18:19,910 I 539544 539544] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:18:19,910 I 539544 539544] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:18:19,910 I 539544 539544] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:18:19,917 I 539544 539544] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:18:19,919 I 539544 539544] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: 063de515bcd1cfb869334de5a715b5874be5a15719a885c14de92dd2
[2025-01-15 18:18:19,920 I 539544 539544] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:18:19,924 I 539544 539544] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 54927.
[2025-01-15 18:18:20,170 I 539544 539544] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-15 18:18:20,171 I 539544 539544] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 89.875 ms, max = 248.938 ms, min = 3.244 us, total = 2.247 s
Execution time:  mean = 10.042 ms, total = 251.060 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.666 ms, total = 248.990 ms, Queueing time: mean = 192.637 ms, max = 248.169 ms, min = 3.244 us, total = 1.734 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 18.984 us, total = 94.922 us, Queueing time: mean = 105.256 us, max = 113.890 us, min = 92.715 us, total = 526.279 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.833 us, total = 11.334 us, Queueing time: mean = 124.420 ms, max = 248.938 ms, min = 248.743 ms, total = 497.681 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 12.880 us, total = 25.759 us, Queueing time: mean = 6.978 ms, max = 13.671 ms, min = 284.406 us, total = 13.956 ms
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 1.917 ms, total = 1.917 ms, Queueing time: mean = 976.948 us, max = 976.948 us, min = 976.948 us, total = 976.948 us
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 21.045 us, total = 21.045 us, Queueing time: mean = 4.966 us, max = 4.966 us, min = 4.966 us, total = 4.966 us
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:18:20,171 I 539544 539544] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 371.861 us, max = 1.713 ms, min = 5.482 us, total = 1.859 ms
Execution time:  mean = 542.961 us, total = 2.715 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 900.516 us, total = 2.702 ms, Queueing time: mean = 588.855 us, max = 1.713 ms, min = 5.482 us, total = 1.767 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 13.257 us, total = 13.257 us, Queueing time: mean = 92.740 us, max = 92.740 us, min = 92.740 us, total = 92.740 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:18:20,171 I 539544 539544] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 1.576 ms, max = 7.732 ms, min = 10.660 us, total = 7.879 ms
Execution time:  mean = 49.879 us, total = 249.396 us
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 77.936 us, total = 233.809 us, Queueing time: mean = 2.591 ms, max = 7.732 ms, min = 10.660 us, total = 7.774 ms
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 15.587 us, total = 15.587 us, Queueing time: mean = 105.334 us, max = 105.334 us, min = 105.334 us, total = 105.334 us


[2025-01-15 18:18:20,171 I 539544 539544] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 5 total (0 active)
Queueing time: mean = 233.402 us, max = 912.073 us, min = 12.999 us, total = 1.167 ms
Execution time:  mean = 330.541 us, total = 1.653 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 550.134 us, total = 1.650 ms, Queueing time: mean = 325.877 us, max = 912.073 us, min = 12.999 us, total = 977.632 us
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 1.151 us, total = 2.302 us, Queueing time: mean = 94.688 us, max = 97.264 us, min = 92.112 us, total = 189.376 us


[2025-01-15 18:18:21,439 I 539544 539544] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:21,439 I 539544 539544] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:21,439 I 539544 539544] (gcs_server) gcs_placement_group_manager.cc:819: A new node: f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:18:21,446 I 539544 539630] (gcs_server) ray_syncer.cc:377: Get connection node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:22,463 I 539544 539544] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 539477
[2025-01-15 18:18:22,463 I 539544 539544] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 539477
[2025-01-15 18:18:23,069 I 539544 539544] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:18:23,091 I 539544 539544] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,091 I 539544 539544] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,091 I 539544 539544] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,091 I 539544 539544] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,253 I 539544 539593] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,254 I 539544 539593] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=f4a460f269c1b4af2b87c77f6a07dbb7778e8cdb90495adf8ce6844a
[2025-01-15 18:18:23,254 I 539544 539544] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:18:23,255 I 539544 539544] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:18:23,343 I 539544 539544] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:18:23,343 I 539544 539544] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:18:23,415 I 539544 539544] (gcs_server) stats.h:120: Stats module has shutdown.