Hugging Face's logo

JayKimDevolved
/
tmp

Transformers
Safetensors
Generated from Trainer
trl
sft
Inference Endpoints
Model card Files Community
tmp
File size: 11,705 Bytes 
c011401
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
 
[2025-01-15 18:18:39,834 I 543025 543025] (gcs_server) gcs_server_main.cc:52: Ray cluster metadata ray_version=2.40.0 ray_commit=22541c38dbef25286cd6d19f1c151bf4fd62f2ed
[2025-01-15 18:18:39,834 I 543025 543025] (gcs_server) io_service_pool.cc:35: IOServicePool is running with 1 io_service.
[2025-01-15 18:18:39,841 I 543025 543025] (gcs_server) event.cc:493: Ray Event initialized for GCS
[2025-01-15 18:18:39,841 I 543025 543025] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_NODE
[2025-01-15 18:18:39,841 I 543025 543025] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_ACTOR
[2025-01-15 18:18:39,841 I 543025 543025] (gcs_server) event.cc:493: Ray Event initialized for EXPORT_DRIVER_JOB
[2025-01-15 18:18:39,841 I 543025 543025] (gcs_server) event.cc:324: Set ray event level to warning
[2025-01-15 18:18:39,851 I 543025 543025] (gcs_server) gcs_server.cc:73: GCS storage type is StorageType::IN_MEMORY
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:42: Loading job table data.
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:54: Loading node table data.
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:80: Loading actor table data.
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:93: Loading actor task spec table data.
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:66: Loading placement group table data.
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:46: Finished loading job table data, size = 0
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:58: Finished loading node table data, size = 0
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:84: Finished loading actor table data, size = 0
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:97: Finished loading actor task spec table data, size = 0
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_init_data.cc:71: Finished loading placement group table data, size = 0
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_server.cc:162: No existing server cluster ID found. Generating new ID: bef35d938d864e0ee9d5bea039d0487157c9828741be23be5b6bb859
[2025-01-15 18:18:39,852 I 543025 543025] (gcs_server) gcs_server.cc:644: Autoscaler V2 enabled: 0
[2025-01-15 18:18:39,855 I 543025 543025] (gcs_server) grpc_server.cc:134: GcsServer server started, listening on port 58596.
[2025-01-15 18:18:40,103 I 543025 543025] (gcs_server) gcs_server.cc:245: Gcs Debug state:

GcsNodeManager: 
- RegisterNode request count: 0
- DrainNode request count: 0
- GetAllNodeInfo request count: 0

GcsActorManager: 
- RegisterActor request count: 0
- CreateActor request count: 0
- GetActorInfo request count: 0
- GetNamedActorInfo request count: 0
- GetAllActorInfo request count: 0
- KillActor request count: 0
- ListNamedActors request count: 0
- Registered actors count: 0
- Destroyed actors count: 0
- Named actors count: 0
- Unresolved actors count: 0
- Pending actors count: 0
- Created actors count: 0
- owners_: 0
- actor_to_register_callbacks_: 0
- actor_to_restart_callbacks_: 0
- actor_to_create_callbacks_: 0
- sorted_destroyed_actor_list_: 0

GcsResourceManager: 
- GetAllAvailableResources request count: 0
- GetAllTotalResources request count: 0
- GetAllResourceUsage request count: 0

GcsPlacementGroupManager: 
- CreatePlacementGroup request count: 0
- RemovePlacementGroup request count: 0
- GetPlacementGroup request count: 0
- GetAllPlacementGroup request count: 0
- WaitPlacementGroupUntilReady request count: 0
- GetNamedPlacementGroup request count: 0
- Scheduling pending placement group count: 0
- Registered placement groups count: 0
- Named placement group count: 0
- Pending placement groups count: 0
- Infeasible placement groups count: 0

Publisher:

[runtime env manager] ID to URIs table:
[runtime env manager] URIs reference table:

GcsTaskManager: 
-Total num task events reported: 0
-Total num status task events dropped: 0
-Total num profile events dropped: 0
-Current num of task events stored: 0
-Total num of actor creation tasks: 0
-Total num of actor tasks: 0
-Total num of normal tasks: 0
-Total num of driver tasks: 0

GcsAutoscalerStateManager: 
- last_seen_autoscaler_state_version_: 0
- last_cluster_resource_state_version_: 0
- pending demands:



[2025-01-15 18:18:40,104 I 543025 543025] (gcs_server) gcs_server.cc:843: Main service Event stats:


Global stats: 25 total (5 active)
Queueing time: mean = 90.225 ms, max = 249.120 ms, min = 2.244 us, total = 2.256 s
Execution time:  mean = 10.063 ms, total = 251.568 ms
Event stats:
	GcsInMemoryStore.Put - 9 total (0 active), Execution time: mean = 27.683 ms, total = 249.147 ms, Queueing time: mean = 193.126 ms, max = 248.576 ms, min = 2.244 us, total = 1.738 s
	GcsInMemoryStore.GetAll - 5 total (0 active), Execution time: mean = 10.374 us, total = 51.872 us, Queueing time: mean = 69.221 us, max = 74.917 us, min = 63.652 us, total = 346.105 us
	PeriodicalRunner.RunFnPeriodically - 4 total (2 active, 1 running), Execution time: mean = 2.212 us, total = 8.849 us, Queueing time: mean = 124.523 ms, max = 249.120 ms, min = 248.972 ms, total = 498.091 ms
	event_loop_lag_probe - 2 total (0 active), Execution time: mean = 13.380 us, total = 26.761 us, Queueing time: mean = 9.027 ms, max = 17.856 ms, min = 196.598 us, total = 18.053 ms
	NodeInfoGcsService.grpc_server.GetClusterId.HandleRequestImpl - 1 total (0 active), Execution time: mean = 2.319 ms, total = 2.319 ms, Queueing time: mean = 1.011 ms, max = 1.011 ms, min = 1.011 ms, total = 1.011 ms
	NodeInfoGcsService.grpc_server.GetClusterId - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	ClusterResourceManager.ResetRemoteNodeView - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s
	GcsInMemoryStore.Get - 1 total (0 active), Execution time: mean = 14.316 us, total = 14.316 us, Queueing time: mean = 3.463 us, max = 3.463 us, min = 3.463 us, total = 3.463 us
	RayletLoadPulled - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:18:40,104 I 543025 543025] (gcs_server) gcs_server.cc:847: task_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 475.738 us, max = 1.322 ms, min = 10.929 us, total = 2.379 ms
Execution time:  mean = 949.086 us, total = 4.745 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.578 ms, total = 4.733 ms, Queueing time: mean = 763.967 us, max = 1.322 ms, min = 10.929 us, total = 2.292 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 12.454 us, total = 12.454 us, Queueing time: mean = 86.788 us, max = 86.788 us, min = 86.788 us, total = 86.788 us
	GcsTaskManager.GcJobSummary - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:18:40,104 I 543025 543025] (gcs_server) gcs_server.cc:847: pubsub_io_context Event stats:


Global stats: 5 total (1 active)
Queueing time: mean = 1.017 ms, max = 4.615 ms, min = 8.883 us, total = 5.087 ms
Execution time:  mean = 636.887 us, total = 3.184 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.057 ms, total = 3.170 ms, Queueing time: mean = 1.666 ms, max = 4.615 ms, min = 8.883 us, total = 4.997 ms
	PeriodicalRunner.RunFnPeriodically - 1 total (0 active), Execution time: mean = 14.910 us, total = 14.910 us, Queueing time: mean = 90.102 us, max = 90.102 us, min = 90.102 us, total = 90.102 us
	Publisher.CheckDeadSubscribers - 1 total (1 active), Execution time: mean = 0.000 s, total = 0.000 s, Queueing time: mean = 0.000 s, max = -0.000 s, min = 9223372036.855 s, total = 0.000 s


[2025-01-15 18:18:40,104 I 543025 543025] (gcs_server) gcs_server.cc:847: ray_syncer_io_context Event stats:


Global stats: 5 total (0 active)
Queueing time: mean = 615.360 us, max = 2.652 ms, min = 9.196 us, total = 3.077 ms
Execution time:  mean = 842.487 us, total = 4.212 ms
Event stats:
	event_loop_lag_probe - 3 total (0 active), Execution time: mean = 1.403 ms, total = 4.210 ms, Queueing time: mean = 966.540 us, max = 2.652 ms, min = 9.196 us, total = 2.900 ms
	RaySyncerRegister - 2 total (0 active), Execution time: mean = 975.000 ns, total = 1.950 us, Queueing time: mean = 88.589 us, max = 88.738 us, min = 88.440 us, total = 177.178 us


[2025-01-15 18:18:42,413 I 543025 543025] (gcs_server) gcs_node_manager.cc:85: Registering node info, address = 192.168.0.2, node name = 192.168.0.2 node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:42,414 I 543025 543025] (gcs_server) gcs_node_manager.cc:91: Finished registering node info, address = 192.168.0.2, node name = 192.168.0.2, is_head_node = 1 node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:42,414 I 543025 543025] (gcs_server) gcs_placement_group_manager.cc:819: A new node: c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746 registered, will try to reschedule all the infeasible placement groups.
[2025-01-15 18:18:42,420 I 543025 543104] (gcs_server) ray_syncer.cc:377: Get connection node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,334 I 543025 543025] (gcs_server) gcs_job_manager.cc:90: Adding job, job id = 01000000, driver pid = 542958
[2025-01-15 18:18:43,334 I 543025 543025] (gcs_server) gcs_job_manager.cc:111: Finished adding job, job id = 01000000, driver pid = 542958
[2025-01-15 18:18:43,413 I 543025 543025] (gcs_server) gcs_job_manager.cc:149: Finished marking job state, job id = 01000000
[2025-01-15 18:18:43,479 I 543025 543025] (gcs_server) gcs_node_manager.cc:366: Removing node, node name = 192.168.0.2, death reason = EXPECTED_TERMINATION, death message = received SIGTERM node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,479 I 543025 543025] (gcs_server) gcs_placement_group_manager.cc:789: Node failed, rescheduling the placement groups on the dead node. node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,479 I 543025 543025] (gcs_server) gcs_actor_manager.cc:1274: Node failed, reconstructing actors. node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,479 I 543025 543025] (gcs_server) gcs_job_manager.cc:454: Node failed, mark all jobs from this node as finished node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,729 I 543025 543074] (gcs_server) ray_syncer-inl.h:318: Failed to read the message from: c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,729 I 543025 543074] (gcs_server) ray_syncer.cc:373: Connection is broken. node_id=c0d769e5f92d6460660141efcae6a83591d2581c36624fb7c0e31746
[2025-01-15 18:18:43,742 I 543025 543025] (gcs_server) gcs_server_main.cc:130: GCS server received SIGTERM, shutting down...
[2025-01-15 18:18:43,744 I 543025 543025] (gcs_server) gcs_server.cc:267: Stopping GCS server.
[2025-01-15 18:18:43,825 I 543025 543025] (gcs_server) gcs_server.cc:284: GCS server stopped.
[2025-01-15 18:18:43,825 I 543025 543025] (gcs_server) io_service_pool.cc:47: IOServicePool is stopped.
[2025-01-15 18:18:43,849 I 543025 543025] (gcs_server) stats.h:120: Stats module has shutdown.