| 2023-05-25 17:41:28,141 INFO StreamThr :2921999 [internal.py:wandb_internal():86] W&B internal server running at pid: 2921999, started at: 2023-05-25 17:41:28.141410 | |
| 2023-05-25 17:41:28,142 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: status | |
| 2023-05-25 17:41:28,144 INFO WriterThread:2921999 [datastore.py:open_for_write():85] open: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/run-owp5uwvq.wandb | |
| 2023-05-25 17:41:28,144 DEBUG SenderThread:2921999 [sender.py:send():375] send: header | |
| 2023-05-25 17:41:28,144 DEBUG SenderThread:2921999 [sender.py:send():375] send: run | |
| 2023-05-25 17:41:28,687 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-05-25 17:41:28,688 INFO SenderThread:2921999 [dir_watcher.py:__init__():219] watching files in: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files | |
| 2023-05-25 17:41:28,688 INFO SenderThread:2921999 [sender.py:_start_run_threads():1124] run started: owp5uwvq with start time 1685007688.1411 | |
| 2023-05-25 17:41:28,688 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: summary_record | |
| 2023-05-25 17:41:28,688 INFO SenderThread:2921999 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:41:28,688 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: check_version | |
| 2023-05-25 17:41:28,710 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-05-25 17:41:28,712 DEBUG HandlerThread:2921999 [system_info.py:__init__():31] System info init | |
| 2023-05-25 17:41:28,712 DEBUG HandlerThread:2921999 [system_info.py:__init__():46] System info init done | |
| 2023-05-25 17:41:28,712 INFO HandlerThread:2921999 [system_monitor.py:start():181] Starting system monitor | |
| 2023-05-25 17:41:28,712 INFO SystemMonitor:2921999 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-05-25 17:41:28,712 INFO HandlerThread:2921999 [system_monitor.py:probe():201] Collecting system info | |
| 2023-05-25 17:41:28,712 INFO SystemMonitor:2921999 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-05-25 17:41:28,713 INFO SystemMonitor:2921999 [interfaces.py:start():190] Started disk monitoring | |
| 2023-05-25 17:41:28,713 INFO SystemMonitor:2921999 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-05-25 17:41:28,714 INFO SystemMonitor:2921999 [interfaces.py:start():190] Started memory monitoring | |
| 2023-05-25 17:41:28,715 INFO SystemMonitor:2921999 [interfaces.py:start():190] Started network monitoring | |
| 2023-05-25 17:41:28,724 DEBUG HandlerThread:2921999 [system_info.py:probe():195] Probing system | |
| 2023-05-25 17:41:28,726 DEBUG HandlerThread:2921999 [git.py:repo():40] git repository is invalid | |
| 2023-05-25 17:41:28,726 DEBUG HandlerThread:2921999 [system_info.py:probe():240] Probing system done | |
| 2023-05-25 17:41:28,727 DEBUG HandlerThread:2921999 [system_monitor.py:probe():210] {'os': 'Linux-5.10.134-13.1.al8.x86_64-x86_64-with-glibc2.17', 'python': '3.8.16', 'heartbeatAt': '2023-05-25T09:41:28.724050', 'startedAt': '2023-05-25T09:41:28.135634', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--dataset_name=lambdalabs/pokemon-blip-captions', '--dataloader_num_workers=8', '--resolution=512', '--center_crop', '--random_flip', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--max_train_steps=15000', '--learning_rate=1e-04', '--max_grad_norm=1', '--lr_scheduler=cosine', '--lr_warmup_steps=0', '--output_dir=/home/long.qul/tritontest', '--push_to_hub', '--hub_model_id=', '--report_to=wandb', '--checkpointing_steps=500', '--validation_prompt=A pokemon with blue eyes.', '--seed=1337'), 'state': 'running', 'program': 'lora_test_1.py', 'codePath': 'lora_test_1.py', 'host': 'iZt4n6er62uu4xnw6wibnhZ', 'username': 'long.qul', 'executable': '/home/long.qul/miniconda3/envs/triton/bin/python', 'cpu_count': 28, 'cpu_count_logical': 56, 'cpu_freq': {'current': 2899.998000000001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 1968.4237327575684, 'used': 256.4754180908203}, 'gpu': 'NVIDIA A10', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A10', 'memory_total': 23836098560}], 'memory': {'total': 339.9116630554199}} | |
| 2023-05-25 17:41:28,727 INFO HandlerThread:2921999 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-05-25 17:41:28,727 INFO HandlerThread:2921999 [system_monitor.py:probe():214] Publishing system info | |
| 2023-05-25 17:41:28,727 DEBUG HandlerThread:2921999 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-05-25 17:41:28,727 DEBUG HandlerThread:2921999 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-05-25 17:41:28,727 DEBUG HandlerThread:2921999 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment | |
| 2023-05-25 17:41:29,689 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-summary.json | |
| 2023-05-25 17:41:29,689 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/requirements.txt | |
| 2023-05-25 17:41:29,689 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/conda-environment.yaml | |
| 2023-05-25 17:41:30,813 DEBUG HandlerThread:2921999 [system_info.py:_save_conda():86] Saving conda packages done | |
| 2023-05-25 17:41:30,813 INFO HandlerThread:2921999 [system_monitor.py:probe():216] Finished publishing system info | |
| 2023-05-25 17:41:30,817 DEBUG SenderThread:2921999 [sender.py:send():375] send: files | |
| 2023-05-25 17:41:30,818 INFO SenderThread:2921999 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now | |
| 2023-05-25 17:41:30,821 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-05-25 17:41:30,821 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: stop_status | |
| 2023-05-25 17:41:31,097 DEBUG SenderThread:2921999 [sender.py:send():375] send: telemetry | |
| 2023-05-25 17:41:31,097 DEBUG SenderThread:2921999 [sender.py:send():375] send: config | |
| 2023-05-25 17:41:31,098 DEBUG SenderThread:2921999 [sender.py:send():375] send: exit | |
| 2023-05-25 17:41:31,098 INFO SenderThread:2921999 [sender.py:send_exit():598] handling exit code: 1 | |
| 2023-05-25 17:41:31,098 INFO SenderThread:2921999 [sender.py:send_exit():600] handling runtime: 2 | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:send_exit():606] send defer | |
| 2023-05-25 17:41:31,099 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,099 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 0 | |
| 2023-05-25 17:41:31,099 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 0 | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 1 | |
| 2023-05-25 17:41:31,099 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,099 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 1 | |
| 2023-05-25 17:41:31,099 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 1 | |
| 2023-05-25 17:41:31,099 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 2 | |
| 2023-05-25 17:41:31,099 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,099 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 2 | |
| 2023-05-25 17:41:31,099 INFO HandlerThread:2921999 [system_monitor.py:finish():190] Stopping system monitor | |
| 2023-05-25 17:41:31,100 DEBUG SystemMonitor:2921999 [system_monitor.py:_start():159] Starting system metrics aggregation loop | |
| 2023-05-25 17:41:31,100 INFO HandlerThread:2921999 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-05-25 17:41:31,100 DEBUG SystemMonitor:2921999 [system_monitor.py:_start():166] Finished system metrics aggregation loop | |
| 2023-05-25 17:41:31,100 INFO HandlerThread:2921999 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-05-25 17:41:31,100 DEBUG SystemMonitor:2921999 [system_monitor.py:_start():170] Publishing last batch of metrics | |
| 2023-05-25 17:41:31,131 INFO HandlerThread:2921999 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-05-25 17:41:31,131 INFO HandlerThread:2921999 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-05-25 17:41:31,131 INFO HandlerThread:2921999 [interfaces.py:finish():202] Joined network monitor | |
| 2023-05-25 17:41:31,131 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,131 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 2 | |
| 2023-05-25 17:41:31,131 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 3 | |
| 2023-05-25 17:41:31,131 DEBUG SenderThread:2921999 [sender.py:send():375] send: stats | |
| 2023-05-25 17:41:31,131 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,132 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 3 | |
| 2023-05-25 17:41:31,132 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,132 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 3 | |
| 2023-05-25 17:41:31,132 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 4 | |
| 2023-05-25 17:41:31,132 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,132 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 4 | |
| 2023-05-25 17:41:31,132 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,132 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 4 | |
| 2023-05-25 17:41:31,132 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 5 | |
| 2023-05-25 17:41:31,132 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,132 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 5 | |
| 2023-05-25 17:41:31,132 DEBUG SenderThread:2921999 [sender.py:send():375] send: summary | |
| 2023-05-25 17:41:31,132 INFO SenderThread:2921999 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:41:31,133 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,133 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 5 | |
| 2023-05-25 17:41:31,133 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 6 | |
| 2023-05-25 17:41:31,133 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,133 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 6 | |
| 2023-05-25 17:41:31,133 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,133 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 6 | |
| 2023-05-25 17:41:31,135 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-05-25 17:41:31,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-summary.json | |
| 2023-05-25 17:41:31,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/conda-environment.yaml | |
| 2023-05-25 17:41:31,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-metadata.json | |
| 2023-05-25 17:41:31,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/output.log | |
| 2023-05-25 17:41:31,876 INFO wandb-upload_0:2921999 [upload_job.py:push():137] Uploaded file /tmp/tmpdil908p6wandb/9a4zmdsm-wandb-metadata.json | |
| 2023-05-25 17:41:31,982 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 7 | |
| 2023-05-25 17:41:31,982 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:31,983 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 7 | |
| 2023-05-25 17:41:31,983 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:31,983 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 7 | |
| 2023-05-25 17:41:32,086 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:41:32,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/config.yaml | |
| 2023-05-25 17:41:33,104 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 8 | |
| 2023-05-25 17:41:33,104 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:41:33,104 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:33,104 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 8 | |
| 2023-05-25 17:41:33,104 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:33,105 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 8 | |
| 2023-05-25 17:41:33,105 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 9 | |
| 2023-05-25 17:41:33,105 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:33,105 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 9 | |
| 2023-05-25 17:41:33,105 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:33,105 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 9 | |
| 2023-05-25 17:41:33,105 INFO SenderThread:2921999 [dir_watcher.py:finish():365] shutting down directory watcher | |
| 2023-05-25 17:41:33,690 INFO Thread-12 :2921999 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/output.log | |
| 2023-05-25 17:41:33,691 INFO SenderThread:2921999 [dir_watcher.py:finish():395] scan: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files | |
| 2023-05-25 17:41:33,691 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-summary.json wandb-summary.json | |
| 2023-05-25 17:41:33,691 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/requirements.txt requirements.txt | |
| 2023-05-25 17:41:33,691 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/conda-environment.yaml conda-environment.yaml | |
| 2023-05-25 17:41:33,694 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/output.log output.log | |
| 2023-05-25 17:41:33,694 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-metadata.json wandb-metadata.json | |
| 2023-05-25 17:41:33,694 INFO SenderThread:2921999 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/config.yaml config.yaml | |
| 2023-05-25 17:41:33,694 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 10 | |
| 2023-05-25 17:41:33,695 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:33,696 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 10 | |
| 2023-05-25 17:41:33,697 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:33,698 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 10 | |
| 2023-05-25 17:41:33,698 INFO SenderThread:2921999 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:41:34,667 INFO wandb-upload_0:2921999 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/requirements.txt | |
| 2023-05-25 17:41:34,678 INFO wandb-upload_2:2921999 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/conda-environment.yaml | |
| 2023-05-25 17:41:34,779 INFO wandb-upload_1:2921999 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/wandb-summary.json | |
| 2023-05-25 17:41:35,396 INFO wandb-upload_3:2921999 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/output.log | |
| 2023-05-25 17:41:35,604 INFO wandb-upload_4:2921999 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/files/config.yaml | |
| 2023-05-25 17:41:35,804 INFO Thread-11 :2921999 [sender.py:transition_state():626] send defer: 11 | |
| 2023-05-25 17:41:35,804 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:35,805 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 11 | |
| 2023-05-25 17:41:35,805 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:35,805 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 11 | |
| 2023-05-25 17:41:35,805 INFO SenderThread:2921999 [file_pusher.py:join():172] waiting for file pusher | |
| 2023-05-25 17:41:35,805 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 12 | |
| 2023-05-25 17:41:35,805 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:35,805 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 12 | |
| 2023-05-25 17:41:35,805 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:35,805 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 12 | |
| 2023-05-25 17:41:36,057 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 13 | |
| 2023-05-25 17:41:36,057 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:36,057 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 13 | |
| 2023-05-25 17:41:36,058 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:36,058 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 13 | |
| 2023-05-25 17:41:36,058 INFO SenderThread:2921999 [sender.py:transition_state():626] send defer: 14 | |
| 2023-05-25 17:41:36,058 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:41:36,058 INFO HandlerThread:2921999 [handler.py:handle_request_defer():170] handle defer: 14 | |
| 2023-05-25 17:41:36,058 DEBUG SenderThread:2921999 [sender.py:send():375] send: final | |
| 2023-05-25 17:41:36,058 DEBUG SenderThread:2921999 [sender.py:send():375] send: footer | |
| 2023-05-25 17:41:36,058 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:41:36,058 INFO SenderThread:2921999 [sender.py:send_request_defer():622] handle sender defer: 14 | |
| 2023-05-25 17:41:36,059 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:41:36,059 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:41:36,059 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: server_info | |
| 2023-05-25 17:41:36,059 DEBUG SenderThread:2921999 [sender.py:send_request():402] send_request: server_info | |
| 2023-05-25 17:41:36,060 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: get_summary | |
| 2023-05-25 17:41:36,061 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: sampled_history | |
| 2023-05-25 17:41:36,526 INFO MainThread:2921999 [wandb_run.py:_footer_history_summary_info():3469] rendering history | |
| 2023-05-25 17:41:36,526 INFO MainThread:2921999 [wandb_run.py:_footer_history_summary_info():3501] rendering summary | |
| 2023-05-25 17:41:36,526 INFO MainThread:2921999 [wandb_run.py:_footer_sync_info():3428] logging synced files | |
| 2023-05-25 17:41:36,526 DEBUG HandlerThread:2921999 [handler.py:handle_request():144] handle_request: shutdown | |
| 2023-05-25 17:41:36,526 INFO HandlerThread:2921999 [handler.py:finish():842] shutting down handler | |
| 2023-05-25 17:41:37,059 INFO WriterThread:2921999 [datastore.py:close():298] close: /home/long.qul/tritontest/wandb/run-20230525_174128-owp5uwvq/run-owp5uwvq.wandb | |
| 2023-05-25 17:41:37,526 INFO SenderThread:2921999 [sender.py:finish():1550] shutting down sender | |
| 2023-05-25 17:41:37,526 INFO SenderThread:2921999 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:41:37,526 INFO SenderThread:2921999 [file_pusher.py:join():172] waiting for file pusher | |