| 2023-05-25 17:53:52,728 INFO StreamThr :2933662 [internal.py:wandb_internal():86] W&B internal server running at pid: 2933662, started at: 2023-05-25 17:53:52.727782 | |
| 2023-05-25 17:53:52,729 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: status | |
| 2023-05-25 17:53:52,730 INFO WriterThread:2933662 [datastore.py:open_for_write():85] open: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/run-jw7zshqk.wandb | |
| 2023-05-25 17:53:52,731 DEBUG SenderThread:2933662 [sender.py:send():375] send: header | |
| 2023-05-25 17:53:52,731 DEBUG SenderThread:2933662 [sender.py:send():375] send: run | |
| 2023-05-25 17:53:53,258 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: check_version | |
| 2023-05-25 17:53:53,259 INFO SenderThread:2933662 [dir_watcher.py:__init__():219] watching files in: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files | |
| 2023-05-25 17:53:53,259 INFO SenderThread:2933662 [sender.py:_start_run_threads():1124] run started: jw7zshqk with start time 1685008432.727397 | |
| 2023-05-25 17:53:53,259 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: summary_record | |
| 2023-05-25 17:53:53,259 INFO SenderThread:2933662 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:53:53,259 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: check_version | |
| 2023-05-25 17:53:53,282 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: run_start | |
| 2023-05-25 17:53:53,284 DEBUG HandlerThread:2933662 [system_info.py:__init__():31] System info init | |
| 2023-05-25 17:53:53,284 DEBUG HandlerThread:2933662 [system_info.py:__init__():46] System info init done | |
| 2023-05-25 17:53:53,284 INFO HandlerThread:2933662 [system_monitor.py:start():181] Starting system monitor | |
| 2023-05-25 17:53:53,284 INFO SystemMonitor:2933662 [system_monitor.py:_start():145] Starting system asset monitoring threads | |
| 2023-05-25 17:53:53,284 INFO HandlerThread:2933662 [system_monitor.py:probe():201] Collecting system info | |
| 2023-05-25 17:53:53,285 INFO SystemMonitor:2933662 [interfaces.py:start():190] Started cpu monitoring | |
| 2023-05-25 17:53:53,285 INFO SystemMonitor:2933662 [interfaces.py:start():190] Started disk monitoring | |
| 2023-05-25 17:53:53,285 INFO SystemMonitor:2933662 [interfaces.py:start():190] Started gpu monitoring | |
| 2023-05-25 17:53:53,286 INFO SystemMonitor:2933662 [interfaces.py:start():190] Started memory monitoring | |
| 2023-05-25 17:53:53,286 INFO SystemMonitor:2933662 [interfaces.py:start():190] Started network monitoring | |
| 2023-05-25 17:53:53,292 DEBUG HandlerThread:2933662 [system_info.py:probe():195] Probing system | |
| 2023-05-25 17:53:53,295 DEBUG HandlerThread:2933662 [git.py:repo():40] git repository is invalid | |
| 2023-05-25 17:53:53,295 DEBUG HandlerThread:2933662 [system_info.py:probe():240] Probing system done | |
| 2023-05-25 17:53:53,295 DEBUG HandlerThread:2933662 [system_monitor.py:probe():210] {'os': 'Linux-5.10.134-13.1.al8.x86_64-x86_64-with-glibc2.17', 'python': '3.8.16', 'heartbeatAt': '2023-05-25T09:53:53.292165', 'startedAt': '2023-05-25T09:53:52.722001', 'docker': None, 'cuda': None, 'args': ('--pretrained_model_name_or_path=runwayml/stable-diffusion-v1-5', '--dataset_name=lambdalabs/pokemon-blip-captions', '--dataloader_num_workers=8', '--resolution=512', '--center_crop', '--random_flip', '--train_batch_size=1', '--gradient_accumulation_steps=4', '--max_train_steps=15000', '--learning_rate=1e-04', '--max_grad_norm=1', '--lr_scheduler=cosine', '--lr_warmup_steps=0', '--output_dir=/home/long.qul/tritontest', '--push_to_hub', '--hub_model_id=', '--report_to=wandb', '--checkpointing_steps=500', '--validation_prompt=A pokemon with blue eyes.', '--seed=1337'), 'state': 'running', 'program': 'lora_test_1.py', 'codePath': 'lora_test_1.py', 'host': 'iZt4n6er62uu4xnw6wibnhZ', 'username': 'long.qul', 'executable': '/home/long.qul/miniconda3/envs/triton/bin/python', 'cpu_count': 28, 'cpu_count_logical': 56, 'cpu_freq': {'current': 2899.998000000001, 'min': 0.0, 'max': 0.0}, 'cpu_freq_per_core': [{'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}, {'current': 2899.998, 'min': 0.0, 'max': 0.0}], 'disk': {'total': 1968.4237327575684, 'used': 256.4756889343262}, 'gpu': 'NVIDIA A10', 'gpu_count': 1, 'gpu_devices': [{'name': 'NVIDIA A10', 'memory_total': 23836098560}], 'memory': {'total': 339.9116630554199}} | |
| 2023-05-25 17:53:53,295 INFO HandlerThread:2933662 [system_monitor.py:probe():211] Finished collecting system info | |
| 2023-05-25 17:53:53,295 INFO HandlerThread:2933662 [system_monitor.py:probe():214] Publishing system info | |
| 2023-05-25 17:53:53,295 DEBUG HandlerThread:2933662 [system_info.py:_save_pip():51] Saving list of pip packages installed into the current environment | |
| 2023-05-25 17:53:53,295 DEBUG HandlerThread:2933662 [system_info.py:_save_pip():67] Saving pip packages done | |
| 2023-05-25 17:53:53,296 DEBUG HandlerThread:2933662 [system_info.py:_save_conda():74] Saving list of conda packages installed into the current environment | |
| 2023-05-25 17:53:54,260 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/conda-environment.yaml | |
| 2023-05-25 17:53:54,260 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/requirements.txt | |
| 2023-05-25 17:53:54,260 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-summary.json | |
| 2023-05-25 17:53:55,363 DEBUG HandlerThread:2933662 [system_info.py:_save_conda():86] Saving conda packages done | |
| 2023-05-25 17:53:55,364 INFO HandlerThread:2933662 [system_monitor.py:probe():216] Finished publishing system info | |
| 2023-05-25 17:53:55,368 DEBUG SenderThread:2933662 [sender.py:send():375] send: files | |
| 2023-05-25 17:53:55,368 INFO SenderThread:2933662 [sender.py:_save_file():1378] saving file wandb-metadata.json with policy now | |
| 2023-05-25 17:53:55,371 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: stop_status | |
| 2023-05-25 17:53:55,371 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: stop_status | |
| 2023-05-25 17:53:55,619 DEBUG SenderThread:2933662 [sender.py:send():375] send: telemetry | |
| 2023-05-25 17:53:55,619 DEBUG SenderThread:2933662 [sender.py:send():375] send: config | |
| 2023-05-25 17:53:55,646 DEBUG SenderThread:2933662 [sender.py:send():375] send: exit | |
| 2023-05-25 17:53:55,646 INFO SenderThread:2933662 [sender.py:send_exit():598] handling exit code: 1 | |
| 2023-05-25 17:53:55,646 INFO SenderThread:2933662 [sender.py:send_exit():600] handling runtime: 2 | |
| 2023-05-25 17:53:55,646 INFO SenderThread:2933662 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:53:55,647 INFO SenderThread:2933662 [sender.py:send_exit():606] send defer | |
| 2023-05-25 17:53:55,647 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,647 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 0 | |
| 2023-05-25 17:53:55,647 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,647 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 0 | |
| 2023-05-25 17:53:55,647 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 1 | |
| 2023-05-25 17:53:55,647 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,647 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 1 | |
| 2023-05-25 17:53:55,647 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,647 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 1 | |
| 2023-05-25 17:53:55,647 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 2 | |
| 2023-05-25 17:53:55,647 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,647 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 2 | |
| 2023-05-25 17:53:55,647 INFO HandlerThread:2933662 [system_monitor.py:finish():190] Stopping system monitor | |
| 2023-05-25 17:53:55,648 INFO HandlerThread:2933662 [interfaces.py:finish():202] Joined cpu monitor | |
| 2023-05-25 17:53:55,648 DEBUG SystemMonitor:2933662 [system_monitor.py:_start():159] Starting system metrics aggregation loop | |
| 2023-05-25 17:53:55,648 INFO HandlerThread:2933662 [interfaces.py:finish():202] Joined disk monitor | |
| 2023-05-25 17:53:55,648 DEBUG SystemMonitor:2933662 [system_monitor.py:_start():166] Finished system metrics aggregation loop | |
| 2023-05-25 17:53:55,648 DEBUG SystemMonitor:2933662 [system_monitor.py:_start():170] Publishing last batch of metrics | |
| 2023-05-25 17:53:55,679 INFO HandlerThread:2933662 [interfaces.py:finish():202] Joined gpu monitor | |
| 2023-05-25 17:53:55,679 INFO HandlerThread:2933662 [interfaces.py:finish():202] Joined memory monitor | |
| 2023-05-25 17:53:55,679 INFO HandlerThread:2933662 [interfaces.py:finish():202] Joined network monitor | |
| 2023-05-25 17:53:55,679 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,679 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 2 | |
| 2023-05-25 17:53:55,679 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 3 | |
| 2023-05-25 17:53:55,679 DEBUG SenderThread:2933662 [sender.py:send():375] send: stats | |
| 2023-05-25 17:53:55,679 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,680 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 3 | |
| 2023-05-25 17:53:55,680 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 3 | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 4 | |
| 2023-05-25 17:53:55,680 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,680 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 4 | |
| 2023-05-25 17:53:55,680 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 4 | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 5 | |
| 2023-05-25 17:53:55,680 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,680 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 5 | |
| 2023-05-25 17:53:55,680 DEBUG SenderThread:2933662 [sender.py:send():375] send: summary | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:_save_file():1378] saving file wandb-summary.json with policy end | |
| 2023-05-25 17:53:55,680 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 5 | |
| 2023-05-25 17:53:55,680 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 6 | |
| 2023-05-25 17:53:55,681 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,681 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 6 | |
| 2023-05-25 17:53:55,681 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,681 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 6 | |
| 2023-05-25 17:53:55,683 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: status_report | |
| 2023-05-25 17:53:55,984 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 7 | |
| 2023-05-25 17:53:55,984 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:55,984 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 7 | |
| 2023-05-25 17:53:55,984 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:55,984 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 7 | |
| 2023-05-25 17:53:55,988 INFO wandb-upload_0:2933662 [upload_job.py:push():137] Uploaded file /tmp/tmpfd18htn5wandb/qjhthzap-wandb-metadata.json | |
| 2023-05-25 17:53:56,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/conda-environment.yaml | |
| 2023-05-25 17:53:56,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/config.yaml | |
| 2023-05-25 17:53:56,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-summary.json | |
| 2023-05-25 17:53:56,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-metadata.json | |
| 2023-05-25 17:53:56,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_created():278] file/dir created: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/output.log | |
| 2023-05-25 17:53:56,646 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:53:57,626 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 8 | |
| 2023-05-25 17:53:57,626 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:53:57,626 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:57,627 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 8 | |
| 2023-05-25 17:53:57,627 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:57,627 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 8 | |
| 2023-05-25 17:53:57,627 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 9 | |
| 2023-05-25 17:53:57,627 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:57,627 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 9 | |
| 2023-05-25 17:53:57,627 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:57,627 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 9 | |
| 2023-05-25 17:53:57,627 INFO SenderThread:2933662 [dir_watcher.py:finish():365] shutting down directory watcher | |
| 2023-05-25 17:53:57,647 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:53:58,261 INFO Thread-12 :2933662 [dir_watcher.py:_on_file_modified():295] file/dir modified: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/output.log | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():395] scan: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-summary.json wandb-summary.json | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/requirements.txt requirements.txt | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/conda-environment.yaml conda-environment.yaml | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/output.log output.log | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-metadata.json wandb-metadata.json | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [dir_watcher.py:finish():409] scan save: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/config.yaml config.yaml | |
| 2023-05-25 17:53:58,262 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 10 | |
| 2023-05-25 17:53:58,262 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:53:58,266 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:58,266 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 10 | |
| 2023-05-25 17:53:58,268 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:58,269 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 10 | |
| 2023-05-25 17:53:58,269 INFO SenderThread:2933662 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:53:58,648 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:53:58,648 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:53:58,877 INFO wandb-upload_4:2933662 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/config.yaml | |
| 2023-05-25 17:53:58,890 INFO wandb-upload_3:2933662 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/output.log | |
| 2023-05-25 17:53:58,948 INFO wandb-upload_0:2933662 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/wandb-summary.json | |
| 2023-05-25 17:53:58,970 INFO wandb-upload_2:2933662 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/conda-environment.yaml | |
| 2023-05-25 17:53:59,034 INFO wandb-upload_1:2933662 [upload_job.py:push():137] Uploaded file /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/files/requirements.txt | |
| 2023-05-25 17:53:59,234 INFO Thread-11 :2933662 [sender.py:transition_state():626] send defer: 11 | |
| 2023-05-25 17:53:59,234 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:59,235 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 11 | |
| 2023-05-25 17:53:59,235 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:59,235 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 11 | |
| 2023-05-25 17:53:59,235 INFO SenderThread:2933662 [file_pusher.py:join():172] waiting for file pusher | |
| 2023-05-25 17:53:59,235 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 12 | |
| 2023-05-25 17:53:59,235 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:59,235 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 12 | |
| 2023-05-25 17:53:59,235 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:59,235 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 12 | |
| 2023-05-25 17:53:59,488 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 13 | |
| 2023-05-25 17:53:59,488 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:59,488 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 13 | |
| 2023-05-25 17:53:59,488 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:59,488 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 13 | |
| 2023-05-25 17:53:59,489 INFO SenderThread:2933662 [sender.py:transition_state():626] send defer: 14 | |
| 2023-05-25 17:53:59,489 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: defer | |
| 2023-05-25 17:53:59,489 INFO HandlerThread:2933662 [handler.py:handle_request_defer():170] handle defer: 14 | |
| 2023-05-25 17:53:59,489 DEBUG SenderThread:2933662 [sender.py:send():375] send: final | |
| 2023-05-25 17:53:59,489 DEBUG SenderThread:2933662 [sender.py:send():375] send: footer | |
| 2023-05-25 17:53:59,489 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: defer | |
| 2023-05-25 17:53:59,489 INFO SenderThread:2933662 [sender.py:send_request_defer():622] handle sender defer: 14 | |
| 2023-05-25 17:53:59,489 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:53:59,489 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:53:59,490 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: poll_exit | |
| 2023-05-25 17:53:59,490 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: poll_exit | |
| 2023-05-25 17:53:59,490 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: server_info | |
| 2023-05-25 17:53:59,490 DEBUG SenderThread:2933662 [sender.py:send_request():402] send_request: server_info | |
| 2023-05-25 17:53:59,491 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: get_summary | |
| 2023-05-25 17:53:59,492 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: sampled_history | |
| 2023-05-25 17:53:59,969 INFO MainThread:2933662 [wandb_run.py:_footer_history_summary_info():3469] rendering history | |
| 2023-05-25 17:53:59,969 INFO MainThread:2933662 [wandb_run.py:_footer_history_summary_info():3501] rendering summary | |
| 2023-05-25 17:53:59,969 INFO MainThread:2933662 [wandb_run.py:_footer_sync_info():3428] logging synced files | |
| 2023-05-25 17:53:59,969 DEBUG HandlerThread:2933662 [handler.py:handle_request():144] handle_request: shutdown | |
| 2023-05-25 17:53:59,969 INFO HandlerThread:2933662 [handler.py:finish():842] shutting down handler | |
| 2023-05-25 17:54:00,490 INFO WriterThread:2933662 [datastore.py:close():298] close: /home/long.qul/tritontest/wandb/run-20230525_175352-jw7zshqk/run-jw7zshqk.wandb | |
| 2023-05-25 17:54:00,969 INFO SenderThread:2933662 [sender.py:finish():1550] shutting down sender | |
| 2023-05-25 17:54:00,969 INFO SenderThread:2933662 [file_pusher.py:finish():167] shutting down file pusher | |
| 2023-05-25 17:54:00,969 INFO SenderThread:2933662 [file_pusher.py:join():172] waiting for file pusher | |