ray/session_2025-01-21_05-47-33_016171_18344/logs/monitor.log

2025-01-21 05:47:34,430	INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py
2025-01-21 05:47:34,430	INFO monitor.py:689 -- Ray version: 2.40.0
2025-01-21 05:47:34,430	INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed
2025-01-21 05:47:34,430	INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-21_05-47-33_016171_18344/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:56605', '--monitor-ip=192.168.0.2']
2025-01-21 05:47:34,448	INFO monitor.py:159 -- session_name: session_2025-01-21_05-47-33_016171_18344
2025-01-21 05:47:34,450	INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217
2025-01-21 05:47:34,451	ERROR monitor.py:208 -- An exception occurred while starting the metrics server.
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py", line 197, in __init__
    prometheus_client.start_http_server(
  File "/usr/local/lib/python3.10/dist-packages/prometheus_client/exposition.py", line 170, in start_wsgi_server
    httpd = make_server(addr, port, app, TmpServer, handler_class=_SilentHandler)
  File "/usr/lib/python3.10/wsgiref/simple_server.py", line 154, in make_server
    server = server_class((host, port), handler_class)
  File "/usr/lib/python3.10/socketserver.py", line 452, in __init__
    self.server_bind()
  File "/usr/lib/python3.10/wsgiref/simple_server.py", line 50, in server_bind
    HTTPServer.server_bind(self)
  File "/usr/lib/python3.10/http/server.py", line 137, in server_bind
    socketserver.TCPServer.server_bind(self)
  File "/usr/lib/python3.10/socketserver.py", line 466, in server_bind
    self.socket.bind(self.server_address)
OSError: [Errno 98] Address already in use
2025-01-21 05:47:34,453	INFO monitor.py:216 -- Monitor: Started
2025-01-21 05:47:34,469	INFO autoscaler.py:280 -- disable_node_updaters:False
2025-01-21 05:47:34,469	INFO autoscaler.py:288 -- disable_launch_config_check:True
2025-01-21 05:47:34,469	INFO autoscaler.py:300 -- foreground_node_launch:False
2025-01-21 05:47:34,469	INFO autoscaler.py:310 -- worker_liveness_check:True
2025-01-21 05:47:34,469	INFO autoscaler.py:318 -- worker_rpc_drain:True
2025-01-21 05:47:34,470	INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []}
2025-01-21 05:47:34,475	INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting.
2025-01-21 05:47:39,486	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:47:39,487	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:47:39.487793 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:47:39,490	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:47:44,499	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:47:44,499	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:47:44.499768 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:47:44,502	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:47:49,511	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:47:49,512	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:47:49.512529 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:47:49,515	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:47:54,524	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:47:54,524	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:47:54.524786 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:47:54,527	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:47:59,537	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:47:59,538	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:47:59.538130 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:47:59,541	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:48:04,551	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:04,552	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:04.552375 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:04,554	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:09,563	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:09,563	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:09.563742 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:09,565	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:48:14,576	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:14,577	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:14.577283 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:14,579	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:19,590	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:19,591	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:19.590807 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:19,593	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:48:24,605	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:24,606	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:24.606436 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:24,609	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:48:29,619	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:29,620	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:29.620625 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:29,623	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:34,634	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:34,634	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:34.634657 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:34,637	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:39,647	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:39,647	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:39.647712 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:39,650	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:44,659	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:44,660	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:44.660357 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:44,663	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:49,670	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:49,670	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:49.670662 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:49,672	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:54,684	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:54,685	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:54.685029 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:54,687	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:48:59,694	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:48:59,695	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:48:59.694841 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:48:59,696	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:04,707	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:04,708	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:04.708200 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:04,710	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:09,719	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:09,720	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:09.720419 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:09,722	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:14,734	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:14,735	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:14.735384 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:14,737	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:19,746	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:19,747	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:19.747483 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:19,749	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:24,759	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:24,760	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:24.760110 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:24,762	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:29,773	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:29,773	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:29.773553 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:29,776	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:34,787	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:34,788	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:34.788390 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:34,791	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:39,802	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:39,802	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:39.802873 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:39,804	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:49:44,817	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:44,818	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:44.817773 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:44,820	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:49:49,831	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:49,832	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:49.832014 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:49,834	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:49:54,843	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:54,843	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:54.843816 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:54,845	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:49:59,851	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:49:59,852	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:49:59.852016 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:49:59,854	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:04,864	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:04,865	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:04.864916 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:04,867	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:09,880	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:09,881	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:09.881314 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:09,884	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:50:14,896	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:14,897	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:14.896939 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:14,899	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:19,912	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:19,913	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:19.912790 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:19,915	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:24,923	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:24,923	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:24.923649 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:24,925	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:29,936	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:29,937	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:29.936882 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:29,939	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:34,947	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:34,948	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:34.948575 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:34,950	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:39,961	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:39,961	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:39.961666 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:39,963	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:44,973	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:44,974	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:44.973938 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:44,976	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:49,985	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:49,986	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:49.986497 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:49,988	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:50:54,998	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:50:54,998	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:50:54.998750 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:50:55,000	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:51:00,010	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:00,011	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:00.011200 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:00,013	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:51:05,023	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:05,024	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:05.024129 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:05,025	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:10,033	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:10,034	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:10.034445 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:10,036	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:15,047	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:15,048	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:15.047922 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:15,049	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:20,058	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:20,059	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:20.059652 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:20,061	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:25,070	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:25,070	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:25.070680 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:25,072	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:51:30,080	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:30,081	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:30.081477 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:30,083	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:51:35,090	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:35,091	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:35.091422 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:35,093	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:40,103	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:40,103	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:40.103667 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:40,105	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:51:45,116	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:45,116	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:45.116770 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:45,119	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:51:50,130	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:50,131	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:50.130926 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:50,132	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:51:55,144	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:51:55,145	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:51:55.145104 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:51:55,147	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:00,156	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:00,157	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:00.157353 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:00,159	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:05,171	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:05,172	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:05.172514 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:05,175	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:10,183	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:10,184	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:10.184561 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:10,187	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:15,199	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:15,200	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:15.200262 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:15,203	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:20,213	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:20,214	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:20.214571 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:20,217	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:25,227	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:25,228	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:25.228274 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:25,230	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:30,243	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:30,244	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:30.244035 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:30,246	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:35,255	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:35,256	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:35.256424 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:35,259	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:40,270	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:40,271	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:40.271091 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:40,273	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:45,283	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:45,284	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:45.284052 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:45,286	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:50,296	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:50,297	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:50.297529 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:50,300	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:52:55,311	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:52:55,312	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:52:55.312351 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:52:55,315	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:00,323	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:00,324	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:00.324360 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:00,326	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:05,339	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:05,340	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:05.340402 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:05,343	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:53:10,355	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:10,356	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:10.356588 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:10,359	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:15,371	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:15,372	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:15.371929 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:15,374	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:20,386	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:20,387	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:20.387208 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:20,390	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:53:25,402	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:25,403	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:25.403177 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:25,405	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:30,415	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:30,416	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:30.416387 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:30,418	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:35,430	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:35,430	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:35.430667 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:35,433	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:40,444	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:40,445	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:40.445275 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:40,447	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:45,459	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:45,460	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:45.460143 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:45,462	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:50,473	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:50,474	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:50.474384 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:50,477	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:53:55,489	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:53:55,490	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:53:55.490404 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:53:55,492	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:00,505	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:00,505	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:00.505779 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:00,508	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:05,520	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:05,521	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:05.521322 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:05,524	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:10,532	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:10,533	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:10.533542 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:10,535	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:15,545	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:15,546	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:15.546109 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:15,548	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:20,558	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:20,559	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:20.559578 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:20,561	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:25,573	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:25,574	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:25.574541 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:25,577	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:54:30,587	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:30,588	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:30.588209 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:30,590	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:35,603	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:35,604	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:35.603866 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:35,606	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:54:40,615	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:40,616	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:40.616549 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:40,619	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:45,631	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:45,632	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:45.632325 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:45,634	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:50,647	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:50,647	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:50.647657 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:50,650	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:54:55,662	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:54:55,663	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:54:55.663320 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:54:55,666	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:55:00,675	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:00,676	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:00.676636 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:00,679	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:05,690	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:05,690	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:05.690751 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:05,693	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:10,703	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:10,704	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:10.704501 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:10,707	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:55:15,720	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:15,721	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:15.720846 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:15,723	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:55:20,735	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:20,736	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:20.736630 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:20,739	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:25,746	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:25,746	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:25.746817 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:25,748	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:55:30,759	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:30,759	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:30.759577 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:30,761	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:55:35,771	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:35,772	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:35.772176 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:35,774	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:40,783	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:40,784	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:40.784015 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:40,786	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:45,795	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:45,795	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:45.795456 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:45,797	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:55:50,807	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:50,808	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:50.808458 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:50,811	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:55:55,823	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:55:55,824	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:55:55.824222 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:55:55,826	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:00,837	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:00,837	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:00.837700 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:00,839	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:05,851	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:05,852	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:05.852435 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:05,855	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:10,863	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:10,864	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:10.864501 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:10,867	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:15,878	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:15,879	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:15.879499 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:15,882	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:20,893	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:20,894	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:20.893813 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:20,896	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:25,906	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:25,907	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:25.907328 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:25,909	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:30,916	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:30,917	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:30.917441 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:30,919	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:35,932	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:35,933	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:35.932920 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:35,935	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:56:40,948	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:40,949	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:40.949088 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:40,951	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:45,962	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:45,963	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:45.963098 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:45,965	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:50,973	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:50,974	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:50.974536 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:50,976	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:56:55,985	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:56:55,986	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:56:55.986465 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:56:55,988	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:00,998	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:00,998	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:00.998867 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:01,000	INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration.
2025-01-21 05:57:06,011	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:06,012	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:06.012176 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:06,014	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:11,025	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:11,026	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:11.026314 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:11,029	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:16,039	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:16,040	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:16.039851 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:16,042	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:21,053	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:21,054	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:21.054571 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:21,057	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:26,069	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:26,070	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:26.069801 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:26,072	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:31,083	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:31,085	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:31.084797 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:31,088	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:57:36,099	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:36,100	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:36.100358 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:36,103	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:57:41,111	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:41,112	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:41.112215 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:41,114	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:46,126	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:46,127	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:46.127304 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:46,129	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:57:51,139	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:51,140	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:51.140624 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:51,143	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-21 05:57:56,155	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:57:56,156	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:57:56.155994 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:57:56,158	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-21 05:58:01,167	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-21 05:58:01,168	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-21 05:58:01.168493 ========
Node status
---------------------------------------------------------------
Active:
 1 node_381e636a10e4140b2e9620d2650d6a018da067c3591f2305edfa793d
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/20.0 CPU
 0.0/2.0 GPU
 0B/70.04GiB memory
 0B/2.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-21 05:58:01,171	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.