|
2025-01-20 22:49:26,963 INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py |
|
2025-01-20 22:49:26,963 INFO monitor.py:689 -- Ray version: 2.40.0 |
|
2025-01-20 22:49:26,963 INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
2025-01-20 22:49:26,963 INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-20_22-49-26_045015_8700/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:44001', '--monitor-ip=192.168.0.2'] |
|
2025-01-20 22:49:26,986 INFO monitor.py:159 -- session_name: session_2025-01-20_22-49-26_045015_8700 |
|
2025-01-20 22:49:26,989 INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217 |
|
2025-01-20 22:49:26,990 INFO monitor.py:216 -- Monitor: Started |
|
2025-01-20 22:49:27,007 INFO autoscaler.py:280 -- disable_node_updaters:False |
|
2025-01-20 22:49:27,007 INFO autoscaler.py:288 -- disable_launch_config_check:True |
|
2025-01-20 22:49:27,007 INFO autoscaler.py:300 -- foreground_node_launch:False |
|
2025-01-20 22:49:27,007 INFO autoscaler.py:310 -- worker_liveness_check:True |
|
2025-01-20 22:49:27,007 INFO autoscaler.py:318 -- worker_rpc_drain:True |
|
2025-01-20 22:49:27,009 INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []} |
|
2025-01-20 22:49:27,013 INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting. |
|
2025-01-20 22:49:32,024 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:32,024 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:32.024610 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:32,026 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:49:37,035 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:37,035 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:37.035715 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:37,038 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:49:42,049 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:42,050 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:42.049957 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:42,052 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:49:47,064 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:47,064 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:47.064729 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:47,067 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:49:52,079 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:52,080 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:52.080212 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:52,083 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:49:57,095 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:49:57,096 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:49:57.096531 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:49:57,099 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:02,112 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:02,112 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:02.112701 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:02,115 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:50:07,128 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:07,129 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:07.129224 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:07,131 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:12,143 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:12,144 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:12.144048 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:12,146 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:17,156 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:17,157 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:17.157452 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:17,160 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:22,171 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:22,172 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:22.172299 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:22,175 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:27,187 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:27,188 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:27.188218 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:27,190 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:32,199 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:32,199 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:32.199642 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:32,201 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-20 22:50:37,213 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:37,214 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:37.214145 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:37,216 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:50:42,229 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:42,230 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:42.230603 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:42,233 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:50:47,245 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:47,246 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:47.246486 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:47,249 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:50:52,261 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:52,262 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:52.262604 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:52,265 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:50:57,275 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:50:57,276 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:50:57.276231 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:50:57,278 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:02,291 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:02,292 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:02.291997 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:02,294 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:07,307 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:07,308 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:07.308249 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:07,311 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:51:12,323 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:12,323 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:12.323716 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:12,326 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:17,335 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:17,336 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:17.336297 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:17,338 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:22,350 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:22,351 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:22.351486 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:22,354 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:27,366 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:27,367 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:27.367444 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:27,370 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:51:32,382 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:32,383 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:32.383580 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:32,386 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:51:37,395 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:37,396 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:37.396469 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:37,399 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:42,407 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:42,408 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:42.408104 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:42,410 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:47,421 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:47,421 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:47.421722 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:47,424 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:51:52,436 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:52,436 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:52.436688 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:52,439 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:51:57,446 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:51:57,446 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:51:57.446579 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:51:57,448 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:52:02,459 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:02,460 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:02.460482 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:02,463 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:07,476 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:07,477 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:07.477126 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:07,479 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:12,493 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:12,494 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:12.493836 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:12,496 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:17,510 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:17,511 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:17.510807 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:17,513 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:22,524 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:22,524 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:22.524509 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:22,526 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:52:27,536 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:27,537 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:27.536836 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:27,539 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:32,551 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:32,551 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:32.551753 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:32,553 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-20 22:52:37,566 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:37,567 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:37.566972 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:37,570 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:42,579 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:42,580 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:42.580668 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:42,583 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:47,595 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:47,596 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:47.595878 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:47,598 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:52,611 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:52,612 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:52.612448 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:52,615 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-20 22:52:57,628 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-20 22:52:57,629 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-20 22:52:57.629236 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_436871bdc85bdc6a74a720eb95141ed73f9ec5b33bf63663a74609e3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/16.0 CPU |
|
0.0/2.0 GPU |
|
0B/53.82GiB memory |
|
0B/26.91GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-20 22:52:57,632 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
|
