|
2025-01-21 05:58:19,543 INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py |
|
2025-01-21 05:58:19,543 INFO monitor.py:689 -- Ray version: 2.40.0 |
|
2025-01-21 05:58:19,543 INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
2025-01-21 05:58:19,543 INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-21_05-58-18_727454_20269/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:58630', '--monitor-ip=192.168.0.2'] |
|
2025-01-21 05:58:19,555 INFO monitor.py:159 -- session_name: session_2025-01-21_05-58-18_727454_20269 |
|
2025-01-21 05:58:19,557 INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217 |
|
2025-01-21 05:58:19,557 ERROR monitor.py:208 -- An exception occurred while starting the metrics server. |
|
Traceback (most recent call last): |
|
File "/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py", line 197, in __init__ |
|
prometheus_client.start_http_server( |
|
File "/usr/local/lib/python3.10/dist-packages/prometheus_client/exposition.py", line 170, in start_wsgi_server |
|
httpd = make_server(addr, port, app, TmpServer, handler_class=_SilentHandler) |
|
File "/usr/lib/python3.10/wsgiref/simple_server.py", line 154, in make_server |
|
server = server_class((host, port), handler_class) |
|
File "/usr/lib/python3.10/socketserver.py", line 452, in __init__ |
|
self.server_bind() |
|
File "/usr/lib/python3.10/wsgiref/simple_server.py", line 50, in server_bind |
|
HTTPServer.server_bind(self) |
|
File "/usr/lib/python3.10/http/server.py", line 137, in server_bind |
|
socketserver.TCPServer.server_bind(self) |
|
File "/usr/lib/python3.10/socketserver.py", line 466, in server_bind |
|
self.socket.bind(self.server_address) |
|
OSError: [Errno 98] Address already in use |
|
2025-01-21 05:58:19,558 INFO monitor.py:216 -- Monitor: Started |
|
2025-01-21 05:58:19,575 INFO autoscaler.py:280 -- disable_node_updaters:False |
|
2025-01-21 05:58:19,575 INFO autoscaler.py:288 -- disable_launch_config_check:True |
|
2025-01-21 05:58:19,575 INFO autoscaler.py:300 -- foreground_node_launch:False |
|
2025-01-21 05:58:19,575 INFO autoscaler.py:310 -- worker_liveness_check:True |
|
2025-01-21 05:58:19,575 INFO autoscaler.py:318 -- worker_rpc_drain:True |
|
2025-01-21 05:58:19,576 INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []} |
|
2025-01-21 05:58:19,579 INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting. |
|
2025-01-21 05:58:24,590 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:24,591 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:24.591098 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:24,593 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 05:58:29,602 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:29,602 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:29.602890 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:29,604 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 05:58:34,615 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:34,616 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:34.616215 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:34,618 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:58:39,629 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:39,630 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:39.629913 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:39,631 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 05:58:44,639 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:44,640 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:44.640546 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:44,643 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:58:49,654 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:49,655 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:49.655332 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:49,657 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:58:54,668 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:54,669 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:54.669665 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:54,672 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:58:59,684 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:58:59,684 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:58:59.684683 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:58:59,687 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:04,696 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:04,696 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:04.696820 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:04,699 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:09,711 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:09,711 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:09.711573 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:09,713 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 05:59:14,724 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:14,724 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:14.724702 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:14,727 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:19,734 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:19,735 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:19.734833 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:19,736 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:24,747 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:24,747 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:24.747703 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:24,749 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:29,759 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:29,759 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:29.759844 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:29,761 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 05:59:34,771 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:34,772 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:34.772418 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:34,774 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:39,783 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:39,784 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:39.784023 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:39,786 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:44,793 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:44,794 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:44.794212 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:44,796 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:49,805 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:49,806 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:49.806313 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:49,808 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:54,816 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:54,817 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:54.816957 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:54,819 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 05:59:59,828 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 05:59:59,829 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 05:59:59.829555 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 05:59:59,831 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:04,838 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:04,839 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:04.839508 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:04,841 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:09,852 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:09,853 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:09.853145 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:09,855 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:14,863 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:14,864 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:14.864073 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:14,866 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:19,875 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:19,876 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:19.876655 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:19,879 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:24,892 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:24,893 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:24.892767 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:24,895 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:00:29,907 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:29,907 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:29.907825 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:29,909 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:00:34,920 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:34,920 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:34.920728 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:34,923 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:00:39,935 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:39,936 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:39.936490 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:39,938 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:44,950 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:44,951 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:44.951688 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:44,954 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:00:49,967 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:49,968 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:49.967941 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:49,970 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:54,982 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:54,983 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:54.983631 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:00:54,986 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:00:59,998 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:00:59,999 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:00:59.999590 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:00,001 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:05,013 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:05,014 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:05.014241 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:05,016 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:10,029 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:10,029 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:10.029710 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:10,032 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:15,044 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:15,045 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:15.045115 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:15,047 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:20,059 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:20,059 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:20.059853 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:20,062 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:25,074 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:25,075 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:25.075197 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:25,078 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-21 06:01:30,085 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:30,086 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:30.086557 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:30,088 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:01:35,098 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:35,099 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:35.099095 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:35,101 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:40,112 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:40,112 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:40.112606 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:40,114 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:45,125 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:45,126 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:45.126441 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:45,128 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:50,139 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:50,140 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:50.140177 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:50,142 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:01:55,152 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:01:55,152 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:01:55.152843 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:01:55,154 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:02:00,162 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:00,163 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:00.163491 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:00,165 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:02:05,173 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:05,173 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:05.173452 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:05,174 INFO autoscaler.py:470 -- The autoscaler took 0.001 seconds to complete the update iteration. |
|
2025-01-21 06:02:10,183 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:10,184 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:10.184127 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:10,186 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:02:15,194 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:15,194 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:15.194668 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:15,196 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:02:20,201 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:20,202 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:20.202543 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:20,204 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-21 06:02:25,214 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:25,214 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:25.214730 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:25,216 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:02:30,226 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:30,227 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:30.226856 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:30,229 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-21 06:02:35,239 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-21 06:02:35,240 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-21 06:02:35.239865 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_331bf2382daee9e1be6a8d365bb49297cc7c87f5fbcfb7e5998c831c |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/20.0 CPU |
|
0.0/2.0 GPU |
|
0B/74.26GiB memory |
|
0B/2.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-21 06:02:35,241 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
|
