|
2025-01-17 16:50:18,578 INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py |
|
2025-01-17 16:50:18,578 INFO monitor.py:689 -- Ray version: 2.40.0 |
|
2025-01-17 16:50:18,579 INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
2025-01-17 16:50:18,579 INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-17_16-50-17_632823_138/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:58609', '--monitor-ip=192.168.0.2'] |
|
2025-01-17 16:50:18,600 INFO monitor.py:159 -- session_name: session_2025-01-17_16-50-17_632823_138 |
|
2025-01-17 16:50:18,602 INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217 |
|
2025-01-17 16:50:18,604 INFO monitor.py:216 -- Monitor: Started |
|
2025-01-17 16:50:18,621 INFO autoscaler.py:280 -- disable_node_updaters:False |
|
2025-01-17 16:50:18,622 INFO autoscaler.py:288 -- disable_launch_config_check:True |
|
2025-01-17 16:50:18,622 INFO autoscaler.py:300 -- foreground_node_launch:False |
|
2025-01-17 16:50:18,622 INFO autoscaler.py:310 -- worker_liveness_check:True |
|
2025-01-17 16:50:18,622 INFO autoscaler.py:318 -- worker_rpc_drain:True |
|
2025-01-17 16:50:18,623 INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []} |
|
2025-01-17 16:50:18,628 INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting. |
|
2025-01-17 16:50:23,651 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:23,652 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:23.652236 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
0B/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:23,655 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 16:50:28,667 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:28,668 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:28.668419 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
10.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
0B/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:28,671 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 16:50:33,679 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:33,679 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:33.679706 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
38.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
168.08MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:33,682 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 16:50:38,691 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:38,692 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:38.692601 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.83MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:38,695 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 16:50:43,703 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:43,704 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:43.704659 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
9.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
216.40MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:43,707 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 16:50:48,720 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:48,721 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:48.721503 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
3.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.87MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 16:50:48,724 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 16:50:53,736 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:53,736 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:53.736748 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
2.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.87MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 16:50:53,740 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 16:50:58,751 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:50:58,752 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:50:58.751874 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
6.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.87MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:50:58,754 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 16:51:03,763 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:51:03,764 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:51:03.763954 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.91MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 16:51:03,765 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-17 16:51:08,774 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 16:51:08,775 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 16:51:08.775119 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_4416fb69be66184a743b5a29c5acc080e3cce7862d0ac53342db7115 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.98MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 16:51:08,777 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
|
