| 2023-05-25 17:40:38,404 INFO StreamThr :2918590 [internal.py:wandb_internal():86] W&B internal server running at pid: 2918590, started at: 2023-05-25 17:40:38.403997 | |
| 2023-05-25 17:40:38,405 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: status | |
| 2023-05-25 17:40:38,407 INFO WriterThread:2918590 [datastore.py:open_for_write():85] open: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/run-2hxthfpt.wandb | |
| 2023-05-25 17:40:38,407 DEBUG SenderThread:2918590 [sender.py:send():375] send: header | |
| 2023-05-25 17:40:38,407 DEBUG SenderThread:2918590 [sender.py:send():375] send: run | |
| 2023-05-25 17:40:39,037 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-05-25 17:40:39,037 INFO SenderThread:2918590 [dir_watcher.py:__init__():219] watching files in: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files | |
| 2023-05-25 17:40:39,037 INFO SenderThread:2918590 [sender.py:_start_run_threads():1124] run started: 2hxthfpt with start time 1685007638.403607 | |
| 2023-05-25 17:40:39,037 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: summary_record | |
| 2023-05-25 17:40:39,037 INFO SenderThread:2918590 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:40:39,037 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: check_version | |
| 2023-05-25 17:40:39,063 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-05-25 17:40:39,065 DEBUG HandlerThread:2918590 [system_info.py:__init__():31] System info init | |
| 2023-05-25 17:40:39,065 DEBUG HandlerThread:2918590 [system_info.py:__init__():46] System info init done | |
| 2023-05-25 17:40:39,065 INFO HandlerThread:2918590 [system_monitor.py:start():181] Starting system monitor | |
| 2023-05-25 17:40:39,065 INFO SystemMonitor:2918590 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-05-25 17:40:39,065 INFO HandlerThread:2918590 [system_monitor.py:probe():201] Collecting system info | |
| 2023-05-25 17:40:39,066 INFO SystemMonitor:2918590 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-05-25 17:40:39,066 INFO SystemMonitor:2918590 [interfaces.py:start():190] Started disk monitoring | |
| 2023-05-25 17:40:39,067 INFO SystemMonitor:2918590 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-05-25 17:40:39,067 INFO SystemMonitor:2918590 [interfaces.py:start():190] Started memory monitoring | |
| 2023-05-25 17:40:39,067 INFO SystemMonitor:2918590 [interfaces.py:start():190] Started network monitoring | |
| 2023-05-25 17:40:39,072 DEBUG HandlerThread:2918590 [system_info.py:probe():195] Probing system | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [git.py:repo():40] git repository is invalid | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [system_info.py:probe():240] Probing system done | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [system_monitor.py:probe():210] {'os': 'Linux-5.10.134-13.1.al8.x86_64-x86_64-with-glibc2.17', 'python': '3.8.16', 'heartbeatAt': '2023-05-25T09:40:39.072595', 'startedAt': '2023-05-25T09:40:38.398394', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--dataset_name=lambdalabs/pokemon-blip-captions', '--dataloader_num_workers=8', '--resolution=512', '--center_crop', '--random_flip', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--max_train_steps=15000', '--learning_rate=1e-04', '--max_grad_norm=1', '--lr_scheduler=cosine', '--lr_warmup_steps=0', '--output_dir=/home/long.qul/tritontest', '--push_to_hub', '--hub_model_id=', '--report_to=wandb', '--checkpointing_steps=500', '--validation_prompt=A pokemon with blue eyes.', '--seed=1337'), 'state': 'running', 'program': 'lora_test_1.py', 'codePath': 'lora_test_1.py', 'host': 'iZt4n6er62uu4xnw6wibnhZ', 'username': 'long.qul', 'executable': '/home/long.qul/miniconda3/envs/triton/bin/python', 'cpu_count': 28, 'cpu_count_logical': 56, 'cpu_freq': {'current': 2899.998000000001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 1968.4237327575684, 'used': 256.4753112792969}, 'gpu': 'NVIDIA A10', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A10', 'memory_total': 23836098560}], 'memory': {'total': 339.9116630554199}} | |
| 2023-05-25 17:40:39,075 INFO HandlerThread:2918590 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-05-25 17:40:39,075 INFO HandlerThread:2918590 [system_monitor.py:probe():214] Publishing system info | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-05-25 17:40:39,075 DEBUG HandlerThread:2918590 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment | |
| 2023-05-25 17:40:40,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/requirements.txt | |
| 2023-05-25 17:40:40,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/conda-environment.yaml | |
| 2023-05-25 17:40:40,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-summary.json | |
| 2023-05-25 17:40:41,149 DEBUG HandlerThread:2918590 [system_info.py:_save_conda():86] Saving conda packages done | |
| 2023-05-25 17:40:41,149 INFO HandlerThread:2918590 [system_monitor.py:probe():216] Finished publishing system info | |
| 2023-05-25 17:40:41,153 DEBUG SenderThread:2918590 [sender.py:send():375] send: files | |
| 2023-05-25 17:40:41,154 INFO SenderThread:2918590 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now | |
| 2023-05-25 17:40:41,157 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-05-25 17:40:41,157 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: stop_status | |
| 2023-05-25 17:40:41,399 DEBUG SenderThread:2918590 [sender.py:send():375] send: telemetry | |
| 2023-05-25 17:40:41,399 DEBUG SenderThread:2918590 [sender.py:send():375] send: config | |
| 2023-05-25 17:40:41,419 DEBUG SenderThread:2918590 [sender.py:send():375] send: exit | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:send_exit():598] handling exit code: 1 | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:send_exit():600] handling runtime: 2 | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:send_exit():606] send defer | |
| 2023-05-25 17:40:41,420 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,420 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 0 | |
| 2023-05-25 17:40:41,420 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 0 | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 1 | |
| 2023-05-25 17:40:41,420 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,420 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 1 | |
| 2023-05-25 17:40:41,420 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 1 | |
| 2023-05-25 17:40:41,420 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 2 | |
| 2023-05-25 17:40:41,421 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,421 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 2 | |
| 2023-05-25 17:40:41,421 INFO HandlerThread:2918590 [system_monitor.py:finish():190] Stopping system monitor | |
| 2023-05-25 17:40:41,421 DEBUG SystemMonitor:2918590 [system_monitor.py:_start():159] Starting system metrics aggregation loop | |
| 2023-05-25 17:40:41,421 INFO HandlerThread:2918590 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-05-25 17:40:41,421 DEBUG SystemMonitor:2918590 [system_monitor.py:_start():166] Finished system metrics aggregation loop | |
| 2023-05-25 17:40:41,421 INFO HandlerThread:2918590 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-05-25 17:40:41,421 DEBUG SystemMonitor:2918590 [system_monitor.py:_start():170] Publishing last batch of metrics | |
| 2023-05-25 17:40:41,451 INFO HandlerThread:2918590 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-05-25 17:40:41,452 INFO HandlerThread:2918590 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-05-25 17:40:41,452 INFO HandlerThread:2918590 [interfaces.py:finish():202] Joined network monitor | |
| 2023-05-25 17:40:41,452 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,452 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 2 | |
| 2023-05-25 17:40:41,452 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 3 | |
| 2023-05-25 17:40:41,452 DEBUG SenderThread:2918590 [sender.py:send():375] send: stats | |
| 2023-05-25 17:40:41,452 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,452 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 3 | |
| 2023-05-25 17:40:41,452 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 3 | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 4 | |
| 2023-05-25 17:40:41,453 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,453 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 4 | |
| 2023-05-25 17:40:41,453 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 4 | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 5 | |
| 2023-05-25 17:40:41,453 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,453 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 5 | |
| 2023-05-25 17:40:41,453 DEBUG SenderThread:2918590 [sender.py:send():375] send: summary | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:40:41,453 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 5 | |
| 2023-05-25 17:40:41,453 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 6 | |
| 2023-05-25 17:40:41,453 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,453 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 6 | |
| 2023-05-25 17:40:41,453 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,454 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 6 | |
| 2023-05-25 17:40:41,456 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-05-25 17:40:41,756 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 7 | |
| 2023-05-25 17:40:41,756 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:41,756 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 7 | |
| 2023-05-25 17:40:41,756 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:41,757 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 7 | |
| 2023-05-25 17:40:42,037 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-summary.json | |
| 2023-05-25 17:40:42,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/conda-environment.yaml | |
| 2023-05-25 17:40:42,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/config.yaml | |
| 2023-05-25 17:40:42,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/output.log | |
| 2023-05-25 17:40:42,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-metadata.json | |
| 2023-05-25 17:40:42,063 INFO wandb-upload_0:2918590 [upload_job.py:push():137] Uploaded file /tmp/tmpzpltiz4xwandb/6hasxp1e-wandb-metadata.json | |
| 2023-05-25 17:40:42,420 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:43,406 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 8 | |
| 2023-05-25 17:40:43,406 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:43,406 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:43,406 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 8 | |
| 2023-05-25 17:40:43,406 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:43,407 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 8 | |
| 2023-05-25 17:40:43,407 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 9 | |
| 2023-05-25 17:40:43,407 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:43,407 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 9 | |
| 2023-05-25 17:40:43,407 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:43,407 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 9 | |
| 2023-05-25 17:40:43,407 INFO SenderThread:2918590 [dir_watcher.py:finish():365] shutting down directory watcher | |
| 2023-05-25 17:40:43,420 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:44,038 INFO Thread-12 :2918590 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/output.log | |
| 2023-05-25 17:40:44,038 INFO SenderThread:2918590 [dir_watcher.py:finish():395] scan: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files | |
| 2023-05-25 17:40:44,038 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-summary.json wandb-summary.json | |
| 2023-05-25 17:40:44,039 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/requirements.txt requirements.txt | |
| 2023-05-25 17:40:44,039 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/conda-environment.yaml conda-environment.yaml | |
| 2023-05-25 17:40:44,039 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/output.log output.log | |
| 2023-05-25 17:40:44,039 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-metadata.json wandb-metadata.json | |
| 2023-05-25 17:40:44,039 INFO SenderThread:2918590 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/config.yaml config.yaml | |
| 2023-05-25 17:40:44,042 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 10 | |
| 2023-05-25 17:40:44,042 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:44,042 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:44,042 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 10 | |
| 2023-05-25 17:40:44,044 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:44,045 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 10 | |
| 2023-05-25 17:40:44,045 INFO SenderThread:2918590 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:40:44,420 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:44,421 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:44,808 INFO wandb-upload_1:2918590 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/requirements.txt | |
| 2023-05-25 17:40:45,110 INFO wandb-upload_0:2918590 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/wandb-summary.json | |
| 2023-05-25 17:40:45,421 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:45,421 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:45,695 INFO wandb-upload_2:2918590 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/conda-environment.yaml | |
| 2023-05-25 17:40:45,725 INFO wandb-upload_4:2918590 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/config.yaml | |
| 2023-05-25 17:40:46,105 INFO wandb-upload_3:2918590 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/files/output.log | |
| 2023-05-25 17:40:46,306 INFO Thread-11 :2918590 [sender.py:transition_state():626] send defer: 11 | |
| 2023-05-25 17:40:46,306 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:46,306 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 11 | |
| 2023-05-25 17:40:46,306 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:46,306 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 11 | |
| 2023-05-25 17:40:46,306 INFO SenderThread:2918590 [file_pusher.py:join():172] waiting for file pusher | |
| 2023-05-25 17:40:46,306 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 12 | |
| 2023-05-25 17:40:46,306 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:46,306 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 12 | |
| 2023-05-25 17:40:46,306 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:46,307 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 12 | |
| 2023-05-25 17:40:46,421 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:46,831 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 13 | |
| 2023-05-25 17:40:46,831 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:46,831 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:46,831 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 13 | |
| 2023-05-25 17:40:46,832 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:46,832 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 13 | |
| 2023-05-25 17:40:46,832 INFO SenderThread:2918590 [sender.py:transition_state():626] send defer: 14 | |
| 2023-05-25 17:40:46,832 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:40:46,832 INFO HandlerThread:2918590 [handler.py:handle_request_defer():170] handle defer: 14 | |
| 2023-05-25 17:40:46,832 DEBUG SenderThread:2918590 [sender.py:send():375] send: final | |
| 2023-05-25 17:40:46,832 DEBUG SenderThread:2918590 [sender.py:send():375] send: footer | |
| 2023-05-25 17:40:46,832 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:40:46,832 INFO SenderThread:2918590 [sender.py:send_request_defer():622] handle sender defer: 14 | |
| 2023-05-25 17:40:46,832 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:46,832 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:46,833 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:40:46,833 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:40:46,833 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: server_info | |
| 2023-05-25 17:40:46,833 DEBUG SenderThread:2918590 [sender.py:send_request():402] send_request: server_info | |
| 2023-05-25 17:40:46,834 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: get_summary | |
| 2023-05-25 17:40:46,835 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: sampled_history | |
| 2023-05-25 17:40:47,297 INFO MainThread:2918590 [wandb_run.py:_footer_history_summary_info():3469] rendering history | |
| 2023-05-25 17:40:47,298 INFO MainThread:2918590 [wandb_run.py:_footer_history_summary_info():3501] rendering summary | |
| 2023-05-25 17:40:47,298 INFO MainThread:2918590 [wandb_run.py:_footer_sync_info():3428] logging synced files | |
| 2023-05-25 17:40:47,298 DEBUG HandlerThread:2918590 [handler.py:handle_request():144] handle_request: shutdown | |
| 2023-05-25 17:40:47,298 INFO HandlerThread:2918590 [handler.py:finish():842] shutting down handler | |
| 2023-05-25 17:40:47,833 INFO WriterThread:2918590 [datastore.py:close():298] close: /home/long.qul/tritontest/wandb/run-20230525_174038-2hxthfpt/run-2hxthfpt.wandb | |
| 2023-05-25 17:40:48,297 INFO SenderThread:2918590 [sender.py:finish():1550] shutting down sender | |
| 2023-05-25 17:40:48,298 INFO SenderThread:2918590 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:40:48,298 INFO SenderThread:2918590 [file_pusher.py:join():172] waiting for file pusher | |