ray/session_2025-01-17_17-06-14_562693_9501/logs/monitor.log

2025-01-17 17:06:15,485	INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py
2025-01-17 17:06:15,486	INFO monitor.py:689 -- Ray version: 2.40.0
2025-01-17 17:06:15,486	INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed
2025-01-17 17:06:15,486	INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-17_17-06-14_562693_9501/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:53056', '--monitor-ip=192.168.0.2']
2025-01-17 17:06:15,504	INFO monitor.py:159 -- session_name: session_2025-01-17_17-06-14_562693_9501
2025-01-17 17:06:15,507	INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217
2025-01-17 17:06:15,508	INFO monitor.py:216 -- Monitor: Started
2025-01-17 17:06:15,526	INFO autoscaler.py:280 -- disable_node_updaters:False
2025-01-17 17:06:15,526	INFO autoscaler.py:288 -- disable_launch_config_check:True
2025-01-17 17:06:15,526	INFO autoscaler.py:300 -- foreground_node_launch:False
2025-01-17 17:06:15,526	INFO autoscaler.py:310 -- worker_liveness_check:True
2025-01-17 17:06:15,526	INFO autoscaler.py:318 -- worker_rpc_drain:True
2025-01-17 17:06:15,527	INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []}
2025-01-17 17:06:15,531	INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting.
2025-01-17 17:06:20,551	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:20,552	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:20.552673 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 0.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 0B/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:06:20,555	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-17 17:06:25,567	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:25,567	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:25.567607 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 10.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 0B/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:06:25,569	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:06:30,580	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:30,581	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:30.581168 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 10.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 76.15MiB/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:06:30,583	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:06:35,652	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:35,653	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:35.653137 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 64.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 189.13MiB/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:06:35,655	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:06:40,667	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:40,667	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:40.667719 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 3.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 147.84MiB/4.00GiB object_store_memory

Demands:
 {'CPU': 1}: 116+ from request_resources()
2025-01-17 17:06:40,671	INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration.
2025-01-17 17:06:45,683	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:45,684	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:45.684277 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 1.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 147.86MiB/4.00GiB object_store_memory

Demands:
 {'CPU': 1}: 116+ from request_resources()
2025-01-17 17:06:45,686	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:06:50,695	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:50,696	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:50.696021 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 1.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 147.85MiB/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:06:50,698	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:06:55,706	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:06:55,707	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:06:55.707466 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 7.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 216.42MiB/4.00GiB object_store_memory

Demands:
 {'CPU': 1}: 116+ from request_resources()
2025-01-17 17:06:55,710	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:07:00,718	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:07:00,719	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:07:00.719485 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 2.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 166.80MiB/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:07:00,721	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.
2025-01-17 17:07:05,731	INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes.
2025-01-17 17:07:05,732	INFO autoscaler.py:427 -- 
======== Autoscaler status: 2025-01-17 17:07:05.732493 ========
Node status
---------------------------------------------------------------
Active:
 1 node_1cc7243d4d7faf0b5672664c331eda22d6e6a5d17cce88079d187efc
Pending:
 (no pending nodes)
Recent failures:
 (no failures)

Resources
---------------------------------------------------------------
Usage:
 1.0/96.0 CPU
 0.0/2.0 GPU
 0B/2.00GiB memory
 147.92MiB/4.00GiB object_store_memory

Demands:
 (no resource demands)
2025-01-17 17:07:05,734	INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration.