|
2025-01-17 17:07:57,716 INFO monitor.py:688 -- Starting monitor using ray installation: /usr/local/lib/python3.10/dist-packages/ray/__init__.py |
|
2025-01-17 17:07:57,717 INFO monitor.py:689 -- Ray version: 2.40.0 |
|
2025-01-17 17:07:57,717 INFO monitor.py:690 -- Ray commit: 22541c38dbef25286cd6d19f1c151bf4fd62f2ed |
|
2025-01-17 17:07:57,717 INFO monitor.py:691 -- Monitor started with command: ['/usr/local/lib/python3.10/dist-packages/ray/autoscaler/_private/monitor.py', '--logs-dir=/tmp/ray/session_2025-01-17_17-07-56_786065_18763/logs', '--logging-rotate-bytes=536870912', '--logging-rotate-backup-count=5', '--gcs-address=192.168.0.2:62173', '--monitor-ip=192.168.0.2'] |
|
2025-01-17 17:07:57,735 INFO monitor.py:159 -- session_name: session_2025-01-17_17-07-56_786065_18763 |
|
2025-01-17 17:07:57,737 INFO monitor.py:191 -- Starting autoscaler metrics server on port 44217 |
|
2025-01-17 17:07:57,738 INFO monitor.py:216 -- Monitor: Started |
|
2025-01-17 17:07:57,755 INFO autoscaler.py:280 -- disable_node_updaters:False |
|
2025-01-17 17:07:57,756 INFO autoscaler.py:288 -- disable_launch_config_check:True |
|
2025-01-17 17:07:57,756 INFO autoscaler.py:300 -- foreground_node_launch:False |
|
2025-01-17 17:07:57,756 INFO autoscaler.py:310 -- worker_liveness_check:True |
|
2025-01-17 17:07:57,756 INFO autoscaler.py:318 -- worker_rpc_drain:True |
|
2025-01-17 17:07:57,757 INFO autoscaler.py:368 -- StandardAutoscaler: {'cluster_name': 'default', 'max_workers': 0, 'upscaling_speed': 1.0, 'docker': {}, 'idle_timeout_minutes': 0, 'provider': {'type': 'readonly', 'use_node_id_as_ip': True, 'disable_launch_config_check': True}, 'auth': {}, 'available_node_types': {'ray.head.default': {'resources': {}, 'node_config': {}, 'max_workers': 0}}, 'head_node_type': 'ray.head.default', 'file_mounts': {}, 'cluster_synced_files': [], 'file_mounts_sync_continuously': False, 'rsync_exclude': [], 'rsync_filter': [], 'initialization_commands': [], 'setup_commands': [], 'head_setup_commands': [], 'worker_setup_commands': [], 'head_start_ray_commands': [], 'worker_start_ray_commands': []} |
|
2025-01-17 17:07:57,761 INFO monitor.py:383 -- Autoscaler has not yet received load metrics. Waiting. |
|
2025-01-17 17:08:02,851 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:02,852 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:02.852127 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
0.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
0B/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:02,854 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 17:08:07,866 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:07,867 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:07.867059 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
10.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
0B/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:07,869 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 17:08:12,880 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:12,881 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:12.881012 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
10.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
114.77MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:12,883 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 17:08:17,896 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:17,896 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:17.896713 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
21.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.82MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 17:08:17,899 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 17:08:22,908 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:22,909 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:22.908882 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.84MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:22,911 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 17:08:27,924 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:27,925 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:27.925003 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.86MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 17:08:27,928 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 17:08:32,940 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:32,941 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:32.941783 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
4.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.87MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
{'CPU': 1}: 116+ from request_resources() |
|
2025-01-17 17:08:32,945 INFO autoscaler.py:470 -- The autoscaler took 0.004 seconds to complete the update iteration. |
|
2025-01-17 17:08:37,955 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:37,956 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:37.956577 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
163.99MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:37,959 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
2025-01-17 17:08:42,966 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:42,967 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:42.967147 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
10.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
166.78MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:42,968 INFO autoscaler.py:470 -- The autoscaler took 0.002 seconds to complete the update iteration. |
|
2025-01-17 17:08:47,979 INFO autoscaler.py:147 -- The autoscaler took 0.0 seconds to fetch the list of non-terminated nodes. |
|
2025-01-17 17:08:47,979 INFO autoscaler.py:427 -- |
|
======== Autoscaler status: 2025-01-17 17:08:47.979826 ======== |
|
Node status |
|
--------------------------------------------------------------- |
|
Active: |
|
1 node_bc155692311b7210365069542359d85e197da50bc2668443d689f677 |
|
Pending: |
|
(no pending nodes) |
|
Recent failures: |
|
(no failures) |
|
|
|
Resources |
|
--------------------------------------------------------------- |
|
Usage: |
|
1.0/96.0 CPU |
|
0.0/2.0 GPU |
|
0B/2.00GiB memory |
|
147.91MiB/4.00GiB object_store_memory |
|
|
|
Demands: |
|
(no resource demands) |
|
2025-01-17 17:08:47,982 INFO autoscaler.py:470 -- The autoscaler took 0.003 seconds to complete the update iteration. |
|
|
