|
2025-01-21 06:16:13,401 INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py |
|
2025-01-21 06:16:13,401 INFO monitor.py:689 -- Ray version: 2.40.0 |
|
2025-01-21 06:16:13,401 INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
2025-01-21 06:16:13,401 INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-21_06-16-12_444635_21900/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:46413', '--monitor-ip=192.168.0.2'] |
|
2025-01-21 06:16:13,418 INFO monitor.py:159 -- session_name: session_2025-01-21_06-16-12_444635_21900 |
|
2025-01-21 06:16:13,420 INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217 |
|
2025-01-21 06:16:13,421 ERROR monitor.py:208 -- An exception occurred while starting the metrics server. |
|
Traceback (most recent call last): |
|
File "/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py", line 197, in __init__ |
|
prometheus_client.start_http_server( |
|
File "/usr/local/lib/python3.10/dist-packages/prometheus_client/exposition.py", line 170, in start_wsgi_server |
|
httpd = make_server(addr, port, app, TmpServer, handler_class=_SilentHandler) |
|
File "/usr/lib/python3.10/wsgiref/simple_server.py", line 154, in make_server |
|
server = server_class((host, port), handler_class) |
|
File "/usr/lib/python3.10/socketserver.py", line 452, in __init__ |
|
self.server_bind() |
|
File "/usr/lib/python3.10/wsgiref/simple_server.py", line 50, in server_bind |
|
HTTPServer.server_bind(self) |
|
File "/usr/lib/python3.10/http/server.py", line 137, in server_bind |
|
socketserver.TCPServer.server_bind(self) |
|
File "/usr/lib/python3.10/socketserver.py", line 466, in server_bind |
|
self.socket.bind(self.server_address) |
|
OSError: [Errno 98] Address already in use |
|
2025-01-21 06:16:13,422 INFO monitor.py:216 -- Monitor: Started |
|
2025-01-21 06:16:13,439 INFO autoscaler.py:280 -- disable_node_updaters:False |
|
2025-01-21 06:16:13,439 INFO autoscaler.py:288 -- disable_launch_config_check:True |
|
2025-01-21 06:16:13,439 INFO autoscaler.py:300 -- foreground_node_launch:False |
|
2025-01-21 06:16:13,440 INFO autoscaler.py:310 -- worker_liveness_check:True |
|
2025-01-21 06:16:13,440 INFO autoscaler.py:318 -- worker_rpc_drain:True |
|
2025-01-21 06:16:13,441 INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []} |
|
2025-01-21 06:16:13,445 INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting. |
|
2025-01-21 06:16:18,455 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:18,456 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:18.456699 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
5.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:18,459 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:16:23,470 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:23,471 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:23.471115 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
4.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:23,474 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:16:28,483 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:28,484 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:28.484647 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
4.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:28,487 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:16:33,495 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:33,496 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:33.495915 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
4.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:33,498 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:16:38,508 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:38,509 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:38.508843 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:38,511 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:16:43,524 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:43,525 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:43.525161 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:43,527 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:16:48,535 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:48,536 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:48.536695 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:48,539 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:16:53,549 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:53,550 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:53.549921 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:53,552 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:16:58,560 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:16:58,561 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:16:58.561080 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:16:58,563 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:17:03,576 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:03,577 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:03.576824 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:03,579 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:08,591 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:08,592 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:08.592696 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:08,595 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:17:13,607 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:13,608 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:13.608368 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:13,611 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:18,623 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:18,624 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:18.624316 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:18,626 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:23,636 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:23,637 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:23.636838 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:23,639 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:17:28,648 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:28,649 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:28.649130 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:28,651 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:33,664 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:33,665 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:33.664802 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:33,667 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:17:38,680 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:38,681 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:38.680949 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:38,683 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:17:43,696 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:43,697 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:43.697139 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:43,699 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:48,709 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:48,710 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:48.709918 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:48,711 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:17:53,721 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:53,722 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:53.721785 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:53,723 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:17:58,730 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:17:58,731 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:17:58.731612 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:17:58,733 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:03,743 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:03,744 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:03.744588 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:03,747 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:08,757 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:08,757 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:08.757581 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:08,759 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:18:13,771 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:13,772 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:13.771856 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:13,774 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:18,785 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:18,786 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:18.786102 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:18,788 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:23,801 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:23,802 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:23.802071 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:23,804 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:28,814 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:28,815 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:28.815042 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:28,817 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:18:33,831 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:33,832 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:33.831855 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:33,834 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:18:38,848 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:38,849 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:38.849179 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:38,852 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:18:43,863 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:43,864 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:43.864524 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:43,867 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:48,879 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:48,880 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:48.880433 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:48,883 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:53,896 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:53,897 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:53.896966 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:53,899 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:18:58,911 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:18:58,911 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:18:58.911771 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:18:58,913 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:03,924 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:03,925 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:03.925250 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:03,927 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:08,937 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:08,938 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:08.937827 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:08,939 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:13,947 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:13,948 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:13.948191 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:13,950 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:18,961 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:18,962 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:18.962656 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
2.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:18,965 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:19:23,978 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:23,979 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:23.979064 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
2.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:23,981 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:28,993 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:28,994 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:28.994012 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
2.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:28,996 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:19:34,008 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:34,009 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:34.008790 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:34,010 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:39,021 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:39,022 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:39.022141 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:39,024 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:44,034 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:44,035 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:44.035208 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:44,037 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:49,044 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:49,044 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:49.044799 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:49,046 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:54,059 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:54,059 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:54.059761 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:54,061 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:19:59,071 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:19:59,072 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:19:59.072321 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:19:59,074 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:04,087 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:04,088 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:04.088563 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:04,091 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:20:09,102 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:09,103 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:09.103326 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:09,105 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:20:14,116 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:14,117 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:14.116960 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:14,119 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:20:19,129 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:19,130 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:19.130166 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:19,132 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:24,143 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:24,144 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:24.143912 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:24,146 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:29,158 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:29,159 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:29.159073 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:29,161 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:20:34,172 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:34,173 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:34.172937 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:34,174 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:39,183 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:39,184 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:39.183893 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:39,186 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:44,196 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:44,197 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:44.197182 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:44,199 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:49,208 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:49,209 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:49.208990 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:49,210 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:20:54,219 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:54,220 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:54.219990 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:54,221 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:20:59,230 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:20:59,231 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:20:59.230886 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:20:59,232 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:21:04,241 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:04,242 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:04.242394 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:04,244 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:21:09,250 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:09,251 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:09.250812 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:09,252 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:21:14,259 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:14,260 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:14.260337 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:14,262 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:19,271 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:19,272 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:19.271966 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:19,274 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:24,286 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:24,286 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:24.286730 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:24,289 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:29,299 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:29,300 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:29.299902 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:29,301 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:21:34,311 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:34,312 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:34.312410 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:34,314 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:39,326 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:39,326 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:39.326778 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:39,329 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:44,339 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:44,340 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:44.340596 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:44,343 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:21:49,355 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:49,356 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:49.356529 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:49,359 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:54,367 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:54,368 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:54.368335 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:54,370 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:21:59,379 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:21:59,380 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:21:59.380321 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:21:59,382 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:22:04,392 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:22:04,393 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:22:04.393149 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:22:04,395 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:22:09,407 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:22:09,408 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:22:09.408241 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:22:09,410 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:22:14,422 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:22:14,423 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:22:14.422886 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:22:14,425 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:22:19,437 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:22:19,438 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:22:19.438471 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_70fc2fa5396fc132250cfaf4b4655b76759ab81263e74a39254c24b3 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/72.42GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:22:19,441 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
|
