2025-01-21 08:33:13,175 INFO StreamThr :287141 [internal.py:wandb_internal():85] W&B internal server running at pid: 287141, started at: 2025-01-21 08:33:13.174243 2025-01-21 08:33:13,177 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status 2025-01-21 08:33:13,186 INFO WriterThread:287141 [datastore.py:open_for_write():87] open: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/run-ma4iz8vp.wandb 2025-01-21 08:33:13,187 DEBUG SenderThread:287141 [sender.py:send():379] send: header 2025-01-21 08:33:13,226 DEBUG SenderThread:287141 [sender.py:send():379] send: run 2025-01-21 08:33:13,469 INFO SenderThread:287141 [dir_watcher.py:__init__():211] watching files in: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files 2025-01-21 08:33:13,469 INFO SenderThread:287141 [sender.py:_start_run_threads():1188] run started: ma4iz8vp with start time 1737448393.177231 2025-01-21 08:33:13,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: check_version 2025-01-21 08:33:13,497 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: check_version 2025-01-21 08:33:13,574 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: run_start 2025-01-21 08:33:13,613 DEBUG HandlerThread:287141 [system_info.py:__init__():26] System info init 2025-01-21 08:33:13,613 DEBUG HandlerThread:287141 [system_info.py:__init__():41] System info init done 2025-01-21 08:33:13,613 INFO HandlerThread:287141 [system_monitor.py:start():194] Starting system monitor 2025-01-21 08:33:13,613 INFO SystemMonitor:287141 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-01-21 08:33:13,613 INFO HandlerThread:287141 [system_monitor.py:probe():214] Collecting system info 2025-01-21 08:33:13,614 INFO SystemMonitor:287141 [interfaces.py:start():188] Started cpu monitoring 2025-01-21 08:33:13,615 INFO SystemMonitor:287141 [interfaces.py:start():188] Started disk monitoring 2025-01-21 08:33:13,616 INFO SystemMonitor:287141 [interfaces.py:start():188] Started gpu monitoring 2025-01-21 08:33:13,616 INFO SystemMonitor:287141 [interfaces.py:start():188] Started memory monitoring 2025-01-21 08:33:13,617 INFO SystemMonitor:287141 [interfaces.py:start():188] Started network monitoring 2025-01-21 08:33:13,670 DEBUG HandlerThread:287141 [system_info.py:probe():152] Probing system 2025-01-21 08:33:13,673 DEBUG HandlerThread:287141 [system_info.py:_probe_git():137] Probing git 2025-01-21 08:33:13,700 DEBUG HandlerThread:287141 [system_info.py:_probe_git():145] Probing git done 2025-01-21 08:33:13,700 DEBUG HandlerThread:287141 [system_info.py:probe():200] Probing system done 2025-01-21 08:33:13,700 DEBUG HandlerThread:287141 [system_monitor.py:probe():223] {'os': 'Linux-6.8.0-48-generic-x86_64-with-glibc2.39', 'python': '3.10.0', 'heartbeatAt': '2025-01-21T08:33:13.670526', 'startedAt': '2025-01-21T08:33:13.158778', 'docker': None, 'cuda': None, 'args': ('--model_family', 'llama', '--apply_instruct_masks', '--token_scaled_loss', '--seq_parallel_size', '8', '--report_to', 'wandb', '--do_train', '--model_name_or_path', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--config_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--tokenizer_name', '/datasets/ai/llama3/meta-llama/models--meta-llama--Meta-Llama-3.1-8B-Instruct/snapshots/5206a32e0bd3067aef1ce90f5528ade7d866253f/', '--run_name', '_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--output_dir', '/scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_', '--config_overrides_json', '', '--gradient_accumulation_steps', '2', '--per_device_train_batch_size', '1', '--bf16', '--learning_rate', '1e-6', '--min_lr_ratio', '0.1', '--lr_scheduler_type', 'cosine', '--max_grad_norm', '1.0', '--adam_beta1', '0.9', '--adam_beta2', '0.95', '--weight_decay', '0.1', '--warmup_ratio', '0.05', '--optim', 'adamw_torch', '--logging_steps', '1', '--log_level', 'info', '--save_steps', '200', '--dataloader_num_workers', '1', '--disable_tqdm', 'true', '--use_fast_tokenizer', 'false', '--remove_unused_columns', 'false', '--ddp_find_unused_parameters', 'false', '--fsdp', 'auto_wrap offload', '--gradient_checkpointing', '--tokenized_mds_train', '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/data/ft/bookclaim_balanced_pack_complete', '--cuda_empty_cache', '--num_train_epochs', '1'), 'state': 'running', 'program': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev/prolong-final/finetune.py', 'codePathLocal': 'finetune.py', 'codePath': 'prolong-final/finetune.py', 'git': {'remote': 'https://github.com/chtmp223/BookGen-dev.git', 'commit': '8724891a13655c832d60eb173f8de20de51c47a3'}, 'email': 'ctpham@umass.edu', 'root': '/work/pi_miyyer_umass_edu/ctpham/BookClaim-dev', 'host': 'gpu021', 'username': 'ctpham_umass_edu', 'executable': '/scratch3/workspace/ctpham_umass_edu-ft/envs/prolong-final/bin/python3.10', 'cpu_count': 112, 'cpu_count_logical': 112, 'cpu_freq': {'current': 894.4125178571427, 'min': 800.0, 'max': 3800.0}, 'cpu_freq_per_core': [{'current': 797.96, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 792.817, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 784.24, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.248, 'min': 800.0, 'max': 3800.0}, {'current': 900.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.741, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.481, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 790.454, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 791.496, 'min': 800.0, 'max': 3800.0}, {'current': 1449.544, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1997.872, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1363.963, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1900.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1836.639, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 1100.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 2007.161, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 3800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 790.206, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 783.079, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}, {'current': 800.0, 'min': 800.0, 'max': 3800.0}], 'disk': {'/': {'total': 438.487850189209, 'used': 16.644447326660156}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 8, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 2015.3287200927734}} 2025-01-21 08:33:13,701 INFO HandlerThread:287141 [system_monitor.py:probe():224] Finished collecting system info 2025-01-21 08:33:13,701 INFO HandlerThread:287141 [system_monitor.py:probe():227] Publishing system info 2025-01-21 08:33:13,701 DEBUG HandlerThread:287141 [system_info.py:_save_conda():209] Saving list of conda packages installed into the current environment 2025-01-21 08:33:14,471 INFO Thread-12 :287141 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/conda-environment.yaml 2025-01-21 08:33:20,472 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/conda-environment.yaml 2025-01-21 08:33:20,533 DEBUG HandlerThread:287141 [system_info.py:_save_conda():224] Saving conda packages done 2025-01-21 08:33:20,536 INFO HandlerThread:287141 [system_monitor.py:probe():229] Finished publishing system info 2025-01-21 08:33:20,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:20,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 08:33:20,544 DEBUG SenderThread:287141 [sender.py:send():379] send: files 2025-01-21 08:33:20,544 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-metadata.json with policy now 2025-01-21 08:33:20,843 INFO wandb-upload_0:287141 [upload_job.py:push():130] Uploaded file /tmp/tmpz4ntsvidwandb/f1n5bvuv-wandb-metadata.json 2025-01-21 08:33:20,848 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: python_packages 2025-01-21 08:33:20,849 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: python_packages 2025-01-21 08:33:20,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:33:20,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:33:20,856 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:21,011 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 08:33:21,012 DEBUG SenderThread:287141 [sender.py:send():379] send: config 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:33:21,023 WARNING SenderThread:287141 [sender.py:send_metric():1405] Seen metric with glob (shouldn't happen) 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 08:33:21,023 DEBUG SenderThread:287141 [sender.py:send():379] send: config 2025-01-21 08:33:21,473 INFO Thread-12 :287141 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-metadata.json 2025-01-21 08:33:21,474 INFO Thread-12 :287141 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/requirements.txt 2025-01-21 08:33:21,474 INFO Thread-12 :287141 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:33:21,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:22,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:23,475 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:33:23,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:24,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:24,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:25,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:26,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:27,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:28,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:29,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:29,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:30,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:31,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:32,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:33,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:34,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:35,484 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:33:35,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:35,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:33:35,849 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:33:35,890 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:36,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:37,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:38,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:39,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:40,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:40,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:41,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:42,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:43,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:44,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:45,856 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:45,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:46,490 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/config.yaml 2025-01-21 08:33:46,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:47,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:48,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:49,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:50,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:33:50,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:33:50,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:51,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:51,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:52,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:53,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:54,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:55,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:56,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:57,859 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:33:57,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:58,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:33:59,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:00,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:01,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:02,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:03,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:34:03,766 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:34:03,767 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:34:03,767 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:34:03,767 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:34:03,767 DEBUG SenderThread:287141 [sender.py:send():379] send: metric 2025-01-21 08:34:03,767 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:34:03,768 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:34:03,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:03,771 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:34:04,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:04,523 INFO Thread-12 :287141 [dir_watcher.py:_on_file_created():271] file/dir created: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:34:04,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:05,524 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:34:05,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:34:05,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:34:05,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:06,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:07,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:08,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:08,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:09,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:10,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:11,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:12,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:13,617 DEBUG SystemMonitor:287141 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-01-21 08:34:13,620 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:34:13,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:14,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:14,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:15,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:16,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:17,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:18,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:19,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:19,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:20,533 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/config.yaml 2025-01-21 08:34:20,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:34:20,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:34:20,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:21,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:22,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:23,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:24,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:25,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:25,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:26,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:27,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:28,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:29,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:30,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:30,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:31,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:32,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:33,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:34,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:35,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:34:35,103 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:34:35,103 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:34:35,109 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:34:35,550 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:34:35,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:34:35,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:34:35,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:35,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:36,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:37,552 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:34:37,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:38,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:39,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:40,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:41,859 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:41,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:42,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:43,622 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:34:43,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:44,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:45,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:46,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:47,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:47,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:48,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:49,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:50,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:34:50,850 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:34:50,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:51,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:52,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:52,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:53,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:54,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:55,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:56,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:57,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:58,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:34:58,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:34:59,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:00,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:01,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:02,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:03,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:03,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:04,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:05,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:35:05,854 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:35:05,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:06,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:35:06,732 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:35:06,733 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:35:06,737 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:35:07,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:07,577 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:35:08,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:09,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:09,578 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:35:09,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:10,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:11,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:12,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:13,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:13,625 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:35:14,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:15,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:15,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:16,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:17,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:18,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:19,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:20,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:20,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:35:20,854 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:35:20,979 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:21,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:22,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:23,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:24,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:25,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:26,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:26,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:27,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:28,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:29,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:30,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:31,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:32,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:32,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:33,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:34,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:35,213 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:35,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:35:35,868 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:35:36,212 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:37,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:37,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:38,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:39,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:40,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:40,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:35:40,882 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:35:40,882 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:35:40,888 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:35:41,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:41,604 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:35:42,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:43,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:43,606 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:35:43,629 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:35:43,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:44,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:45,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:46,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:47,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:48,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:48,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:49,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:50,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:50,857 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:35:50,858 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:35:51,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:52,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:53,435 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:53,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:35:54,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:55,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:56,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:57,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:58,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:59,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:35:59,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:00,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:01,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:02,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:03,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:04,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:04,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:05,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:05,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:36:05,873 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:36:06,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:07,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:08,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:09,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:10,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:10,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:11,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:36:11,224 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:36:11,225 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:36:11,231 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:36:11,632 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:36:11,633 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:36:11,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:12,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:13,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:13,632 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:36:14,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:15,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:15,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:16,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:17,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:18,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:19,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:20,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:20,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:36:20,862 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:36:21,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:21,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:22,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:23,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:24,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:25,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:26,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:26,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:27,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:28,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:29,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:30,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:31,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:32,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:32,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:33,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:34,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:35,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:35,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:36:35,862 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:36:36,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:37,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:38,000 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:38,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:39,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:40,376 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:41,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:42,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:43,381 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:43,637 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:36:43,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:44,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:44,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:36:44,686 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:36:44,686 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:36:44,688 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:36:45,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:45,664 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:36:45,665 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:36:46,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:47,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:48,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:48,875 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:49,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:50,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:50,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:36:50,862 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:36:51,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:52,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:53,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:54,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:36:54,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:55,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:56,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:57,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:58,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:59,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:36:59,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:00,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:01,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:02,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:03,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:04,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:04,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:05,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:05,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:37:05,862 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:37:06,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:07,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:08,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:09,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:09,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:10,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:11,389 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:12,389 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:13,389 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:13,649 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:37:14,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:15,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:15,651 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:15,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:37:15,939 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:37:15,939 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:37:15,940 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:37:16,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:16,692 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:37:17,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:17,693 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:37:18,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:19,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:20,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:20,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:37:20,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:37:20,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:21,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:22,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:23,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:24,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:25,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:26,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:26,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:27,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:28,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:29,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:30,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:31,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:31,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:32,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:33,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:34,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:35,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:35,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:37:35,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:37:36,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:37,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:37,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:38,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:39,508 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:40,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:41,512 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:42,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:42,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:42,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:37:42,985 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:37:42,985 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:37:42,990 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:37:43,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:43,646 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:37:43,720 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:37:43,720 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:37:44,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:45,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:46,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:47,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:47,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:48,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:49,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:50,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:50,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:37:50,862 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:37:51,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:52,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:53,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:53,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:54,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:55,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:56,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:57,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:58,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:37:58,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:37:59,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:00,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:01,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:02,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:03,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:04,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:04,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:05,516 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:05,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:38:05,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:38:06,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:07,527 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:08,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:09,520 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:10,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:10,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:11,527 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:12,527 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:13,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:38:13,030 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:38:13,031 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:38:13,035 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:38:13,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:13,650 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:38:13,750 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:38:13,750 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:38:14,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:15,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:15,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:16,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:17,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:18,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:19,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:20,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:20,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:38:20,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:38:21,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:21,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:22,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:23,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:24,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:25,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:26,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:26,883 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:27,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:28,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:29,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:30,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:31,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:32,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:32,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:33,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:34,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:35,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:35,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:38:35,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:38:36,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:37,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:38,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:38,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:39,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:40,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:41,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:42,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:43,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:43,653 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:38:43,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:44,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:45,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:46,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:47,312 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:38:47,313 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:38:47,313 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:38:47,315 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:38:47,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:47,778 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:38:47,779 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:38:48,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:49,353 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:49,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:50,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:50,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:38:50,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:38:51,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:52,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:53,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:54,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:55,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:38:55,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:56,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:57,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:58,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:38:59,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:00,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:00,889 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:01,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:02,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:03,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:04,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:05,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:05,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:39:05,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:39:05,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:06,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:07,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:08,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:09,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:10,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:11,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:11,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:12,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:13,008 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:39:13,009 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:39:13,010 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:39:13,012 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:39:13,657 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:39:13,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:13,800 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:39:13,800 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:39:14,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:15,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:16,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:17,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:17,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:18,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:19,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:20,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:20,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:39:20,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:39:21,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:22,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:22,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:23,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:24,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:25,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:26,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:27,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:28,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:28,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:29,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:30,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:31,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:32,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:33,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:34,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:34,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:35,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:35,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:39:35,863 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:39:36,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:37,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:38,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:39,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:39,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:40,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:41,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:42,745 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:43,658 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:39:43,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:44,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:39:44,074 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:39:44,074 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:39:44,077 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:39:44,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:44,827 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:39:45,115 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:45,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:45,828 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:39:46,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:47,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:48,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:49,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:50,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:50,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:39:50,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:39:51,015 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:51,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:52,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:53,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:54,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:55,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:56,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:56,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:39:57,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:58,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:39:59,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:00,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:01,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:02,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:02,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:03,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:04,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:05,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:40:05,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:40:05,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:06,887 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:07,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:08,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:08,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:09,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:10,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:11,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:12,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:13,660 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:40:13,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:13,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:40:13,740 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:40:13,741 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:40:13,742 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:40:13,856 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:40:14,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:15,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:15,858 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:40:16,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:17,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:18,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:18,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:19,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:20,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:20,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:40:20,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:40:21,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:22,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:23,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:24,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:24,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:25,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:26,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:27,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:28,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:29,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:29,986 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:30,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:31,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:32,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:33,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:34,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:34,986 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:35,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:35,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:40:35,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:40:36,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:37,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:38,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:39,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:40,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:40,986 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:41,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:42,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:43,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:43,666 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:40:44,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:45,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:45,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:46,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:47,065 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:40:47,066 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:40:47,067 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:40:47,069 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:40:47,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:47,887 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:40:47,887 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:40:48,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:49,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:50,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:50,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:40:50,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:40:51,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:51,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:52,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:53,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:54,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:55,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:56,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:56,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:40:57,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:58,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:40:59,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:00,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:01,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:01,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:02,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:03,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:04,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:05,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:05,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:41:05,864 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:41:06,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:07,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:07,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:08,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:09,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:10,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:11,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:12,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:12,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:13,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:13,671 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:41:14,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:15,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:16,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:17,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:17,781 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:41:17,782 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:41:17,782 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:41:17,784 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:41:17,918 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:41:18,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:18,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:19,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:19,919 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:41:20,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:20,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:41:20,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:41:21,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:22,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:23,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:24,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:24,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:25,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:26,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:27,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:28,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:29,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:29,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:30,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:31,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:32,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:33,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:34,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:34,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:35,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:35,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:41:35,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:41:36,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:37,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:38,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:39,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:40,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:40,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:41,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:42,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:43,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:43,671 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:41:44,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:45,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:45,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:41:45,556 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:41:45,557 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:41:45,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:45,558 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:41:45,945 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:41:45,946 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:41:46,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:47,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:48,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:49,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:50,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:50,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:41:50,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:41:50,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:51,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:52,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:53,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:54,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:55,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:56,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:41:56,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:57,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:58,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:41:59,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:00,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:01,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:01,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:02,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:03,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:04,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:05,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:05,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:42:05,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:42:06,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:06,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:07,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:08,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:09,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:10,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:11,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:12,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:12,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:13,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:13,682 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:42:14,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:14,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:42:14,932 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:42:14,932 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:42:14,933 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:42:14,970 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:42:15,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:15,971 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:42:16,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:17,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:18,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:18,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:19,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:20,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:20,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:42:20,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:42:21,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:22,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:23,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:24,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:24,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:25,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:26,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:27,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:28,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:29,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:30,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:30,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:31,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:32,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:33,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:34,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:35,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:35,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:35,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:42:35,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:42:36,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:37,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:38,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:39,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:40,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:41,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:41,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:42,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:43,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:43,683 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:42:44,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:45,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:46,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:46,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:42:46,989 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:42:46,990 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:42:46,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:46,996 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:42:46,999 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:42:47,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:48,000 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:42:48,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:49,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:50,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:50,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:42:50,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:42:51,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:52,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:52,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:53,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:54,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:55,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:56,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:57,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:58,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:42:58,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:42:59,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:00,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:01,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:02,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:03,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:03,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:04,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:05,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:05,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:43:05,865 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:43:06,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:07,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:08,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:09,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:09,602 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:10,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:11,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:12,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:13,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:13,687 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:43:14,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:14,688 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:15,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:15,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:43:15,972 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:43:15,972 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:43:15,974 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:43:16,025 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:43:16,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:17,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:18,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:19,027 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:43:19,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:20,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:20,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:20,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:43:20,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:43:21,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:22,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:23,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:24,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:25,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:26,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:26,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:27,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:28,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:29,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:30,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:31,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:31,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:32,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:33,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:34,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:35,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:35,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:43:35,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:43:36,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:37,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:37,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:38,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:39,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:40,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:41,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:42,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:42,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:43,696 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:43:43,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:44,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:45,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:46,756 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:47,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:43:47,487 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:43:47,488 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:43:47,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:47,489 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:43:47,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:48,053 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:43:48,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:49,054 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:43:49,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:50,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:43:50,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:43:50,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:51,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:52,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:53,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:53,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:54,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:55,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:56,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:57,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:58,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:43:59,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:43:59,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:00,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:01,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:02,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:03,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:04,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:04,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:05,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:44:05,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:44:05,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:06,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:07,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:08,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:09,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:10,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:10,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:11,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:12,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:13,697 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:44:13,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:14,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:15,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:15,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:16,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:16,977 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:44:16,978 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:44:16,979 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:44:16,980 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:44:17,079 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:44:17,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:18,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:19,081 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:44:19,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:20,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:44:20,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:44:20,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:20,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:21,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:22,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:23,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:24,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:25,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:26,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:26,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:27,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:28,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:29,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:30,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:31,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:31,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:32,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:33,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:34,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:35,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:44:35,866 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:44:35,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:36,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:36,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:37,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:38,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:39,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:40,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:41,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:42,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:42,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:43,701 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:44:43,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:44,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:45,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:46,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:47,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:47,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:48,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:49,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:50,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:44:50,872 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:44:50,966 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:51,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:44:51,749 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:44:51,749 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:44:51,754 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:44:52,107 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:44:52,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:52,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:53,108 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:44:53,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:54,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:55,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:56,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:57,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:58,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:44:58,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:44:59,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:00,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:01,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:02,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:03,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:03,456 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:04,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:05,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:05,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:45:05,868 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:45:06,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:07,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:08,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:08,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:09,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:10,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:11,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:12,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:13,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:13,705 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:45:13,707 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:14,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:15,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:16,232 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:17,232 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:18,232 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:19,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:19,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:20,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:45:20,143 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:45:20,143 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:45:20,144 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:45:20,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:20,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:45:20,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:45:21,131 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:45:21,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:22,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:23,132 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:45:23,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:24,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:25,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:25,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:26,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:27,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:28,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:29,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:30,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:30,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:31,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:32,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:33,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:34,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:35,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:35,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:35,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:45:35,868 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:45:36,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:37,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:38,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:39,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:40,290 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:41,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:41,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:42,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:43,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:43,711 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:45:44,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:45,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:46,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:46,298 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:47,300 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:48,300 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:49,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:45:49,240 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:45:49,241 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:45:49,242 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:45:49,656 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:50,156 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:45:50,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:50,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:45:50,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:45:51,157 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:45:51,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:52,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:52,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:53,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:54,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:55,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:56,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:57,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:58,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:45:58,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:45:59,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:00,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:01,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:02,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:03,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:03,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:04,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:05,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:05,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:46:05,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:46:06,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:07,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:08,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:09,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:09,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:10,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:11,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:12,757 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:13,720 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:46:13,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:14,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:14,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:15,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:16,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:17,642 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:46:17,643 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:46:17,643 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:46:17,645 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:46:18,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:18,179 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:46:18,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:19,180 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:46:19,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:20,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:20,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:20,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:46:20,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:46:21,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:22,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:23,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:24,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:25,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:26,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:26,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:27,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:28,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:29,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:30,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:31,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:32,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:32,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:33,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:34,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:35,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:35,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:46:35,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:46:36,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:37,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:37,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:38,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:39,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:40,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:41,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:42,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:42,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:43,717 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:46:43,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:44,873 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:45,873 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:46,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:47,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:47,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:48,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:49,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:50,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:46:50,507 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:46:50,508 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:46:50,513 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:46:50,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:46:50,939 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:46:50,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:51,212 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:46:51,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:52,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:52,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:53,214 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:46:53,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:54,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:55,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:56,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:57,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:46:57,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:58,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:46:59,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:00,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:01,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:03,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:03,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:04,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:05,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:05,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:47:05,934 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:47:06,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:07,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:08,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:09,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:09,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:10,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:11,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:12,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:13,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:13,721 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:47:14,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:14,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:15,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:16,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:17,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:18,272 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:19,272 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:20,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:20,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:20,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:47:20,938 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:47:21,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:21,674 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:47:21,675 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:47:21,675 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:47:21,681 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:47:22,234 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:47:22,279 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:23,235 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:47:23,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:24,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:25,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:25,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:26,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:27,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:28,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:29,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:30,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:31,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:31,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:32,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:33,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:34,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:35,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:35,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:47:35,938 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:47:36,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:36,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:37,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:38,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:39,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:40,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:41,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:41,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:42,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:43,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:43,725 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:47:44,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:45,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:46,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:46,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:47,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:48,276 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:49,276 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:50,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:50,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:47:50,948 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:47:51,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:51,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:52,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:52,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:47:52,826 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:47:52,827 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:47:52,832 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:47:53,262 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:47:53,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:54,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:55,263 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:47:55,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:56,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:57,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:47:57,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:58,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:47:59,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:00,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:01,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:02,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:03,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:03,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:04,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:05,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:05,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:48:05,941 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:48:06,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:07,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:08,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:09,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:09,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:10,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:11,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:12,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:13,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:13,729 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:48:14,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:14,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:15,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:16,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:17,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:18,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:19,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:20,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:20,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:20,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:48:20,941 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:48:21,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:21,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:48:21,905 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:48:21,905 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:48:21,907 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:48:22,295 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:48:22,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:23,296 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:48:23,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:24,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:25,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:25,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:26,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:27,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:28,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:29,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:30,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:31,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:31,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:32,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:33,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:34,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:35,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:35,941 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:48:35,942 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:48:36,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:36,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:37,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:38,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:39,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:40,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:41,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:42,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:42,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:43,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:43,733 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:48:44,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:45,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:46,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:47,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:47,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:48,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:49,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:50,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:50,941 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:48:50,941 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:48:51,312 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:48:51,314 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:48:51,314 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:48:51,316 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:48:51,318 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:48:51,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:52,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:52,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:53,320 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:48:53,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:54,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:55,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:56,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:57,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:58,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:48:58,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:48:59,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:00,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:01,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:02,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:03,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:04,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:04,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:05,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:05,941 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:49:05,942 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:49:06,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:07,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:08,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:09,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:09,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:10,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:11,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:12,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:13,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:13,738 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:49:14,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:14,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:15,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:16,460 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:17,460 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:18,467 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:19,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:20,175 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:20,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:20,948 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:49:20,951 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:49:21,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:22,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:23,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:23,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:49:23,626 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:49:23,627 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:49:23,629 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:49:24,347 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:49:24,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:25,348 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:49:25,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:25,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:26,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:27,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:28,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:29,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:30,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:31,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:31,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:32,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:33,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:34,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:35,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:35,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:49:35,943 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:49:36,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:37,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:37,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:38,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:39,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:40,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:41,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:42,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:42,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:43,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:43,741 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:49:44,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:45,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:46,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:47,180 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:47,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:48,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:49,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:50,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:50,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:49:50,943 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:49:51,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:52,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:52,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:52,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:49:52,577 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:49:52,578 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:49:52,580 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:49:53,371 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:49:53,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:54,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:55,372 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:49:55,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:56,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:57,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:58,180 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:49:58,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:49:59,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:00,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:01,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:02,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:03,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:04,180 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:04,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:05,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:05,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:50:05,943 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:50:06,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:07,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:08,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:09,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:09,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:10,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:11,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:12,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:13,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:13,745 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:50:14,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:14,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:15,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:16,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:17,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:18,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:19,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:20,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:20,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:20,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:50:20,950 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:50:21,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:22,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:22,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:50:22,801 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:50:22,801 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:50:22,803 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:50:23,400 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:50:23,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:24,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:25,402 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:50:25,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:25,841 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:26,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:27,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:28,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:29,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:30,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:31,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:31,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:32,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:33,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:34,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:35,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:35,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:50:35,946 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:50:36,314 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:36,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:37,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:38,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:39,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:40,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:41,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:42,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:42,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:43,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:43,749 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:50:44,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:45,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:46,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:47,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:48,227 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:48,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:49,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:50,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:50,947 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:50:50,950 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:50:51,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:52,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:53,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:53,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:53,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:50:53,788 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:50:53,789 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:50:53,791 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:50:54,431 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:50:54,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:55,432 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:50:55,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:56,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:57,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:58,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:50:59,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:50:59,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:00,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:01,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:02,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:03,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:04,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:05,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:05,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:05,948 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:51:05,948 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:51:06,500 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:07,500 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:08,500 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:09,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:10,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:10,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:11,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:12,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:13,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:13,753 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:51:14,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:15,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:15,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:16,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:17,512 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:18,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:19,516 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:20,512 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:20,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:51:20,953 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:51:21,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:21,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:22,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:22,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:51:22,997 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:51:22,997 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:51:22,998 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:51:23,458 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:51:23,533 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:24,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:25,459 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:51:25,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:26,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:26,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:27,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:28,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:29,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:30,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:31,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:31,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:32,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:33,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:34,659 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:35,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:35,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:51:35,952 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:51:36,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:36,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:37,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:38,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:39,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:40,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:41,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:41,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:42,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:43,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:43,757 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:51:44,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:45,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:46,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:47,232 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:47,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:48,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:49,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:50,507 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:51:50,509 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:51:50,509 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:51:50,510 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:51:50,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:50,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:51:50,952 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:51:51,487 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:51:51,487 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:51:51,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:52,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:53,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:53,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:54,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:55,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:56,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:57,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:58,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:51:58,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:51:59,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:00,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:01,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:02,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:03,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:04,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:04,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:05,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:05,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:52:05,952 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:52:06,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:07,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:08,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:09,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:10,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:10,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:11,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:12,659 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:13,659 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:13,761 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:52:14,659 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:15,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:15,766 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:16,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:17,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:18,665 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:19,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:20,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:20,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:52:20,956 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:52:21,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:21,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:52:21,215 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:52:21,215 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:52:21,216 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:52:21,511 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:52:21,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:22,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:23,512 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:52:23,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:24,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:25,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:26,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:26,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:27,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:28,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:29,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:30,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:31,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:31,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:33,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:34,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:35,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:35,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:52:35,956 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:52:36,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:36,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:37,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:38,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:39,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:40,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:41,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:42,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:42,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:43,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:43,765 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:52:44,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:45,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:46,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:47,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:47,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:48,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:49,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:50,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:50,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:52:50,713 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:52:50,713 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:52:50,715 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:52:51,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:52:51,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:51,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:52:51,539 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:52:52,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:53,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:53,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:53,541 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:52:54,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:55,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:56,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:57,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:58,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:52:58,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:52:59,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:00,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:01,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:02,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:03,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:04,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:04,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:05,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:06,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:53:06,127 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:53:06,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:07,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:08,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:09,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:10,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:10,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:11,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:12,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:13,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:13,766 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:53:14,139 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:15,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:15,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:16,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:17,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:18,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:18,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:53:18,771 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:53:18,772 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:53:18,777 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:53:19,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:19,568 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:53:20,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:20,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:21,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:53:21,127 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:53:21,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:21,569 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:53:22,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:23,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:24,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:25,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:26,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:26,368 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:27,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:28,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:29,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:30,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:31,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:31,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:32,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:33,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:34,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:35,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:36,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:53:36,127 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:53:36,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:37,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:37,368 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:38,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:39,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:40,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:41,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:42,198 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:42,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:43,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:43,772 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:53:44,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:45,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:46,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:47,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:47,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:53:47,591 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:53:47,591 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:53:47,592 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:47,593 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:53:47,596 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:53:48,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:49,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:49,598 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:53:50,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:51,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:53:51,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:53:51,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:52,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:53,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:53,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:54,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:55,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:56,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:57,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:58,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:53:58,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:53:59,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:00,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:01,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:02,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:03,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:04,283 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:04,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:05,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:06,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:54:06,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:54:06,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:07,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:08,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:09,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:10,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:10,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:11,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:12,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:13,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:13,774 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:54:14,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:15,298 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:15,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:16,294 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:17,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:17,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:54:17,983 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:54:17,984 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:54:17,985 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:54:18,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:18,621 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:54:19,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:19,622 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:54:20,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:21,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:21,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:54:21,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:54:21,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:22,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:23,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:24,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:25,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:26,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:26,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:27,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:28,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:29,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:30,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:31,375 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:31,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:32,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:33,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:34,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:35,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:36,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:54:36,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:54:36,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:37,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:37,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:38,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:39,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:40,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:41,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:42,375 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:42,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:43,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:43,780 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:54:44,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:45,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:46,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:47,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:47,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:48,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:48,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:54:48,850 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:54:48,851 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:54:48,852 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:54:49,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:49,645 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:54:50,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:51,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:54:51,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:54:51,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:51,646 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:54:52,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:53,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:53,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:54,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:55,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:56,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:57,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:58,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:54:59,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:54:59,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:00,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:01,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:02,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:03,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:04,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:05,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:05,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:06,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:55:06,128 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:55:06,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:07,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:08,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:09,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:10,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:11,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:11,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:12,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:13,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:13,782 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:55:14,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:15,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:16,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:16,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:17,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:18,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:19,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:20,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:21,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:55:21,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:55:21,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:22,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:55:22,062 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:55:22,063 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:55:22,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:22,069 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:55:22,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:22,680 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:55:23,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:23,681 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:55:24,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:25,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:26,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:27,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:27,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:28,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:29,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:30,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:31,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:32,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:33,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:33,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:34,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:35,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:36,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:55:36,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:55:36,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:37,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:38,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:38,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:39,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:40,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:41,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:42,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:43,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:43,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:43,786 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:55:44,485 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:45,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:46,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:47,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:48,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:48,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:49,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:49,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:55:49,990 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:55:49,990 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:55:49,992 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:55:50,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:50,707 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:55:51,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:55:51,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:55:51,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:51,708 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:55:52,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:53,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:53,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:54,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:55,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:56,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:57,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:58,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:55:59,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:55:59,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:00,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:01,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:02,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:03,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:04,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:05,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:05,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:06,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:56:06,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:56:06,481 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:07,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:08,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:09,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:10,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:11,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:11,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:12,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:13,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:13,796 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:56:14,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:15,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:16,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:16,494 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:17,501 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:18,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:19,278 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:56:19,280 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:56:19,280 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:56:19,281 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:56:19,748 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:56:19,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:20,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:21,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:56:21,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:56:21,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:21,749 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:56:22,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:22,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:23,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:24,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:25,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:26,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:27,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:27,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:28,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:29,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:30,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:31,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:32,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:33,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:33,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:34,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:35,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:36,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:56:36,131 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:56:36,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:37,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:38,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:38,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:39,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:40,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:41,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:42,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:43,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:43,793 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:56:43,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:44,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:45,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:46,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:47,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:48,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:49,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:49,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:50,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:51,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:56:51,137 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:56:51,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:52,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:56:52,379 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:56:52,379 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:56:52,384 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:56:52,774 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:56:52,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:53,776 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:56:53,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:54,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:56:54,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:55,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:56,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:57,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:58,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:56:59,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:00,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:00,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:01,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:02,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:03,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:04,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:05,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:06,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:57:06,137 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:57:06,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:06,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:07,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:08,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:09,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:10,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:11,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:11,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:12,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:13,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:13,797 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:57:14,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:15,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:16,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:16,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:17,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:18,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:19,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:20,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:21,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:57:21,141 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:57:21,657 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:57:21,658 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:57:21,658 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:57:21,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:21,659 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:57:21,804 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:57:22,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:22,805 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:23,805 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:57:23,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:24,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:25,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:26,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:27,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:27,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:28,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:29,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:30,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:31,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:32,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:33,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:33,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:34,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:35,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:36,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:57:36,141 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:57:36,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:37,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:38,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:39,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:39,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:40,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:41,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:42,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:43,801 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:57:43,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:44,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:44,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:45,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:46,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:47,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:48,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:49,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:50,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:57:50,042 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:57:50,043 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:57:50,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:50,044 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:57:50,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:50,835 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:57:51,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:57:51,141 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:57:51,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:51,836 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:57:52,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:53,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:54,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:55,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:57:55,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:56,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:57,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:58,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:57:59,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:00,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:01,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:01,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:02,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:03,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:04,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:05,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:06,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:58:06,141 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:58:06,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:06,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:07,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:08,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:09,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:10,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:11,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:11,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:12,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:13,805 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:58:13,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:14,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:15,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:16,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:16,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:17,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:18,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:19,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:20,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:21,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:58:21,149 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:58:21,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:21,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:58:21,900 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:58:21,900 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:58:21,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:21,902 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:58:22,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:22,859 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:58:23,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:23,860 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:58:24,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:25,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:26,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:27,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:27,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:28,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:29,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:30,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:31,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:32,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:33,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:33,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:34,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:35,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:36,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:58:36,146 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:58:36,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:37,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:38,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:38,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:39,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:40,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:41,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:42,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:43,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:43,809 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:58:43,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:44,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:45,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:46,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:47,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:48,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:49,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:49,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:50,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:51,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:58:51,146 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:58:51,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:52,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:53,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:54,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:58:54,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:55,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:56,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:57,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:58,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:58:58,049 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:58:58,049 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:58:58,050 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:58:58,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:58,888 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:58:59,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:58:59,889 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:59:00,090 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:00,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:01,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:02,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:03,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:04,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:05,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:05,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:06,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:59:06,146 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:59:06,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:07,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:08,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:09,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:10,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:11,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:11,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:12,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:13,813 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:59:13,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:14,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:15,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:16,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:17,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:17,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:18,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:19,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:20,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:21,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:59:21,146 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:59:21,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:22,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:23,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:23,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:24,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:25,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:26,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:27,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:28,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:28,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:59:28,615 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:59:28,616 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:59:28,617 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:59:28,918 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 08:59:29,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:29,919 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 08:59:30,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:31,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:32,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:33,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:34,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:34,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:35,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:36,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:36,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:59:36,146 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:59:37,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:38,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:39,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:40,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:40,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:41,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:42,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:43,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:43,817 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 08:59:44,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:45,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:45,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:46,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:47,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:48,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:49,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:50,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:51,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:51,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 08:59:51,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 08:59:51,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:52,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:53,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:54,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:55,075 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:56,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:56,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 08:59:57,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:58,076 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:59,076 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 08:59:59,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 08:59:59,920 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 08:59:59,921 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 08:59:59,923 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 08:59:59,957 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:00:00,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:01,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:01,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:01,958 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:00:02,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:03,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:04,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:05,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:06,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:00:06,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:00:06,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:07,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:07,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:08,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:09,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:10,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:11,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:12,444 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:13,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:13,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:13,822 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:00:14,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:15,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:16,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:17,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:18,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:18,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:19,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:20,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:21,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:00:21,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:00:21,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:22,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:23,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:23,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:24,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:25,449 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:26,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:27,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:28,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:29,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:29,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:30,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:30,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:00:30,903 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:00:30,903 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:00:30,904 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:00:30,987 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:00:31,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:31,988 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:00:32,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:33,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:34,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:35,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:35,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:36,146 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:00:36,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:00:36,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:37,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:38,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:39,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:40,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:41,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:41,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:42,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:43,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:43,825 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:00:44,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:45,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:46,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:46,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:47,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:48,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:49,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:50,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:51,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:00:51,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:00:51,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:52,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:52,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:53,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:54,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:55,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:56,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:57,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:00:57,460 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:58,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:00:59,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:00,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:01,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:02,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:02,745 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:01:02,746 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:01:02,747 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:01:02,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:02,748 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:01:03,017 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:01:03,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:04,018 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:01:04,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:05,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:06,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:01:06,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:01:06,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:07,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:08,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:08,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:09,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:10,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:11,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:12,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:13,829 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:01:13,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:13,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:14,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:15,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:16,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:17,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:18,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:19,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:19,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:20,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:21,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:01:21,147 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:01:21,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:22,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:23,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:24,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:24,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:25,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:26,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:27,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:28,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:29,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:30,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:30,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:31,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:32,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:33,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:34,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:35,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:35,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:36,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:01:36,153 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:01:36,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:01:36,518 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:01:36,519 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:01:36,520 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:01:36,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:37,044 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:01:37,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:38,045 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:01:38,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:39,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:40,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:40,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:41,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:42,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:43,833 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:01:43,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:44,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:45,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:45,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:46,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:47,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:48,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:49,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:50,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:51,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:01:51,150 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:01:51,276 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:51,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:52,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:53,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:54,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:55,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:56,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:01:56,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:57,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:58,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:01:59,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:00,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:01,431 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:01,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:02,941 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:03,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:04,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:05,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:06,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:02:06,154 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:02:06,431 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:06,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:07,107 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:02:07,109 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:02:07,109 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:02:07,110 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:02:07,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:08,076 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:02:08,076 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:02:08,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:09,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:10,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:11,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:12,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:12,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:13,837 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:02:13,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:14,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:15,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:16,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:17,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:18,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:18,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:19,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:20,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:21,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:02:21,154 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:02:21,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:22,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:23,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:23,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:24,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:25,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:26,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:27,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:28,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:29,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:29,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:30,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:31,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:32,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:33,948 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:34,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:34,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:35,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:36,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:02:36,156 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:02:36,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:37,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:38,521 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:02:38,522 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:02:38,522 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:02:38,524 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:02:38,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:39,102 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:02:39,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:39,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:40,103 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:02:40,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:41,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:42,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:43,842 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:02:43,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:44,844 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:44,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:45,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:46,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:47,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:48,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:49,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:50,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:50,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:51,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:02:51,156 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:02:51,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:52,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:53,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:54,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:55,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:02:55,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:56,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:57,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:58,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:02:59,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:00,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:00,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:01,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:02,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:03,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:04,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:05,435 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:05,978 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:06,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:03:06,165 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:03:06,979 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:07,979 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:08,979 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:09,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:03:09,007 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:03:09,007 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:03:09,009 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:03:09,131 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:03:09,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:10,132 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:03:10,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:11,047 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:11,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:12,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:13,844 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:03:13,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:14,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:15,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:16,436 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:16,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:17,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:18,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:19,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:20,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:21,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:03:21,159 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:03:21,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:22,436 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:22,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:23,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:24,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:25,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:26,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:27,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:27,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:28,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:29,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:30,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:31,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:32,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:33,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:33,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:34,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:35,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:36,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:03:36,166 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:03:36,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:37,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:38,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:38,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:39,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:40,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:41,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:03:41,416 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:03:41,417 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:03:41,420 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:03:41,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:42,155 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:03:42,156 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:03:42,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:43,848 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:03:43,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:43,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:44,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:45,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:46,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:47,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:48,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:49,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:49,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:50,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:51,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:03:51,163 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:03:51,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:53,080 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:54,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:55,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:55,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:03:56,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:57,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:58,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:03:59,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:00,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:00,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:01,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:02,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:03,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:04,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:05,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:05,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:06,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:06,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:04:06,163 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:04:07,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:08,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:09,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:10,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:10,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:10,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:04:10,648 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:04:10,649 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:04:10,650 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:04:11,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:11,185 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:04:12,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:12,186 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:04:13,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:13,852 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:04:14,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:15,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:15,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:16,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:17,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:18,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:19,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:20,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:21,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:21,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:04:21,163 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:04:21,298 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:22,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:23,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:24,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:25,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:26,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:26,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:27,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:28,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:29,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:30,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:31,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:31,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:32,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:33,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:34,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:35,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:36,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:36,166 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:04:36,166 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:04:36,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:37,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:38,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:39,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:40,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:40,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:04:40,536 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:04:40,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:04:40,537 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:04:41,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:41,214 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:04:42,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:42,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:43,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:43,215 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:04:43,855 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:04:44,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:45,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:46,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:47,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:48,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:48,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:49,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:50,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:51,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:51,166 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:04:51,167 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:04:52,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:53,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:54,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:54,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:04:55,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:56,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:57,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:58,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:59,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:04:59,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:00,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:01,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:02,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:03,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:04,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:04,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:05,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:06,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:06,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:05:06,168 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:05:07,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:08,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:09,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:09,301 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:05:09,302 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:05:09,302 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:05:09,304 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:05:10,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:10,236 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:05:10,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:11,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:11,237 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:05:12,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:13,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:13,859 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:05:14,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:15,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:15,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:16,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:17,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:18,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:19,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:20,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:21,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:21,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:05:21,169 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:05:21,258 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:22,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:23,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:24,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:25,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:26,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:26,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:27,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:28,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:29,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:30,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:31,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:32,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:32,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:33,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:34,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:35,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:36,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:36,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:05:36,176 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:05:37,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:37,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:38,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:38,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:05:38,460 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:05:38,460 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:05:38,462 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:05:39,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:39,262 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:05:40,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:41,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:41,263 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:05:42,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:42,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:43,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:43,864 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:05:44,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:45,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:46,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:47,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:48,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:48,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:49,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:50,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:51,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:51,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:05:51,175 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:05:52,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:53,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:54,160 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:54,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:05:55,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:56,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:57,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:58,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:59,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:05:59,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:00,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:01,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:02,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:03,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:04,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:04,584 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:05,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:06,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:06,179 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:06:06,179 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:06:07,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:07,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:06:07,818 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:06:07,819 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:06:07,820 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:06:08,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:08,295 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:06:09,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:09,296 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:06:09,857 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:10,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:11,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:12,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:13,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:13,867 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:06:14,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:14,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:15,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:16,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:17,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:18,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:19,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:20,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:20,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:21,320 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:06:21,321 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:06:21,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:22,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:23,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:24,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:25,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:26,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:26,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:27,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:28,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:29,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:30,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:31,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:31,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:32,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:33,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:34,320 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:35,322 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:36,227 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:06:36,230 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:06:36,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:36,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:37,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:38,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:39,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:39,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:06:39,728 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:06:39,728 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:06:39,730 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:06:40,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:40,325 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:06:41,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:41,326 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:06:42,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:42,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:43,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:43,871 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:06:44,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:45,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:46,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:47,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:48,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:48,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:49,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:50,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:51,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:06:51,226 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:06:51,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:52,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:53,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:54,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:54,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:06:55,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:56,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:57,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:58,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:06:59,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:00,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:00,584 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:01,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:02,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:03,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:04,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:05,336 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:05,588 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:06,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:07:06,231 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:07:06,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:07,329 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:08,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:09,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:09,660 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:07:09,661 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:07:09,661 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:07:09,662 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:07:10,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:10,355 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:07:10,702 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:11,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:11,356 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:07:12,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:13,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:13,875 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:07:14,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:15,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:15,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:16,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:17,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:18,330 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:19,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:20,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:21,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:07:21,231 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:07:21,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:21,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:22,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:23,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:24,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:25,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:26,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:26,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:27,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:28,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:29,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:30,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:31,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:32,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:32,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:33,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:34,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:35,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:36,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:07:36,231 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:07:36,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:37,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:37,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:38,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:39,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:40,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:40,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:07:40,807 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:07:40,807 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:07:40,809 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:07:41,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:41,380 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:07:42,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:43,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:43,381 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:07:43,595 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:43,879 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:07:44,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:45,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:46,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:47,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:48,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:49,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:49,595 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:50,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:51,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:07:51,231 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:07:51,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:52,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:53,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:54,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:54,596 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:07:55,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:56,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:57,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:58,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:59,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:07:59,596 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:00,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:01,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:02,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:03,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:04,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:05,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:05,596 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:06,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:08:06,231 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:08:06,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:07,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:08,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:09,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:10,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:10,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:11,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:12,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:13,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:13,885 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:08:14,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:14,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:08:14,577 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:08:14,578 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:08:14,580 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:08:15,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:15,409 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:08:15,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:16,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:17,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:17,410 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:08:18,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:19,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:20,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:21,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:08:21,232 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:08:21,328 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:21,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:22,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:23,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:24,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:25,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:26,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:26,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:27,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:28,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:29,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:30,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:31,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:32,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:32,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:33,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:34,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:35,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:36,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:08:36,235 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:08:36,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:37,353 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:37,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:38,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:39,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:40,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:41,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:08:41,119 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:08:41,119 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:08:41,120 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:08:41,434 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:08:41,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:42,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:43,436 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:08:43,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:43,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:43,884 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:08:44,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:45,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:46,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:47,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:48,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:48,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:49,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:50,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:51,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:08:51,232 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:08:51,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:52,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:53,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:54,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:54,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:08:55,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:56,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:57,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:58,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:59,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:08:59,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:00,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:01,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:02,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:03,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:04,602 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:05,602 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:05,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:06,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:09:06,240 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:09:06,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:07,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:08,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:09,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:10,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:10,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:11,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:11,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:09:11,893 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:09:11,894 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:09:11,896 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:09:12,475 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:09:12,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:13,476 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:09:13,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:13,888 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:09:14,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:15,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:15,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:16,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:17,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:18,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:19,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:20,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:21,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:09:21,235 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:09:21,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:21,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:22,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:23,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:24,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:25,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:26,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:26,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:27,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:28,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:29,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:30,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:31,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:32,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:32,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:33,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:34,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:35,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:36,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:09:36,245 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:09:36,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:37,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:37,674 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:38,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:39,674 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:40,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:41,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:09:41,303 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:09:41,304 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:09:41,305 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:09:41,500 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:09:41,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:42,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:42,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:43,501 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:09:43,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:43,893 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:09:44,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:45,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:46,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:47,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:48,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:48,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:49,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:50,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:51,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:09:51,239 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:09:51,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:52,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:53,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:54,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:09:54,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:55,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:56,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:57,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:58,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:09:59,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:00,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:00,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:01,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:02,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:03,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:04,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:05,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:05,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:06,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:10:06,249 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:10:06,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:07,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:08,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:09,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:10,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:10,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:11,408 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:10:11,409 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:10:11,409 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:10:11,410 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:10:11,526 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:10:11,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:12,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:13,528 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:10:13,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:13,896 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:10:14,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:15,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:15,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:16,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:17,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:18,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:19,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:20,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:21,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:10:21,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:10:21,320 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:21,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:22,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:23,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:24,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:25,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:26,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:26,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:27,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:28,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:29,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:30,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:31,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:32,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:32,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:33,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:34,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:35,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:36,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:10:36,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:10:36,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:37,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:38,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:38,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:39,841 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:40,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:41,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:42,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:43,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:43,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:43,904 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:10:44,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:45,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:10:45,042 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:10:45,042 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:10:45,044 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:10:45,553 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:10:45,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:46,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:47,554 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:10:47,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:48,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:49,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:49,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:50,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:51,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:10:51,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:10:51,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:52,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:53,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:54,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:55,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:10:55,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:56,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:57,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:58,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:10:59,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:00,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:01,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:01,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:02,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:03,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:04,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:05,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:06,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:11:06,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:11:06,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:07,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:07,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:08,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:09,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:10,848 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:11,845 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:12,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:12,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:13,859 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:13,902 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:11:14,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:15,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:15,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:11:15,930 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:11:15,930 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:11:15,932 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:11:16,578 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:11:16,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:17,579 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:11:17,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:18,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:18,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:19,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:20,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:21,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:11:21,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:11:21,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:22,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:23,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:24,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:24,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:25,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:26,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:27,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:28,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:29,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:30,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:30,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:31,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:32,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:33,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:34,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:35,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:35,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:36,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:11:36,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:11:36,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:37,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:38,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:39,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:40,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:41,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:41,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:42,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:43,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:43,906 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:11:44,867 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:45,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:46,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:47,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:11:47,054 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:11:47,055 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:11:47,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:47,056 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:11:47,610 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:11:47,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:48,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:49,612 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:11:49,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:50,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:51,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:11:51,247 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:11:51,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:52,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:52,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:53,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:54,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:55,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:56,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:57,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:58,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:11:58,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:11:59,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:00,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:01,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:02,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:03,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:03,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:04,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:05,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:06,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:12:06,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:12:06,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:07,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:08,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:09,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:09,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:10,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:11,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:12,902 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:13,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:13,914 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:12:14,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:14,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:15,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:16,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:12:16,497 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:12:16,498 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:12:16,499 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:12:16,637 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:12:16,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:17,638 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:12:17,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:18,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:19,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:20,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:20,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:21,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:12:21,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:12:21,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:22,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:23,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:24,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:25,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:26,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:26,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:27,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:28,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:29,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:30,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:31,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:31,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:32,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:33,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:34,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:35,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:36,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:12:36,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:12:36,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:37,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:37,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:38,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:39,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:40,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:41,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:42,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:42,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:43,911 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:12:43,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:44,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:45,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:46,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:47,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:47,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:48,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:12:48,356 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:12:48,356 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:12:48,361 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:12:48,670 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:12:48,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:49,671 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:12:49,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:50,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:51,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:12:51,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:12:51,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:52,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:53,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:53,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:54,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:55,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:56,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:57,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:58,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:12:58,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:12:59,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:00,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:01,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:03,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:03,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:04,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:05,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:06,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:13:06,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:13:06,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:07,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:08,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:09,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:09,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:10,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:11,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:12,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:13,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:13,916 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:13:14,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:15,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:15,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:16,257 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:17,263 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:18,260 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:19,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:20,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:20,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:21,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:13:21,248 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:13:21,288 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:21,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:13:21,435 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:13:21,436 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:13:21,441 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:13:21,701 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:13:22,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:23,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:23,702 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:13:24,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:25,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:26,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:26,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:27,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:28,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:29,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:30,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:31,262 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:32,262 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:32,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:33,262 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:34,271 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:35,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:36,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:13:36,249 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:13:36,289 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:37,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:37,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:38,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:39,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:40,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:41,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:42,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:43,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:43,920 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:13:43,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:44,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:45,269 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:46,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:47,275 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:48,281 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:48,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:49,281 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:50,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:51,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:13:51,253 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:13:51,292 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:52,075 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:13:52,076 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:13:52,077 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:13:52,080 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:13:52,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:52,726 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:13:53,494 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:53,727 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:13:54,117 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:13:54,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:55,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:56,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:57,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:58,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:59,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:13:59,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:00,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:01,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:02,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:03,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:04,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:04,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:05,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:06,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:14:06,250 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:14:06,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:07,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:08,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:09,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:10,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:10,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:11,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:12,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:13,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:13,924 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:14:14,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:15,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:15,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:16,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:17,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:18,690 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:19,690 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:20,691 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:20,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:21,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:14:21,257 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:14:21,694 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:22,694 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:23,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:14:23,094 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:14:23,094 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:14:23,098 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:14:23,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:23,751 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:14:24,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:25,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:25,753 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:14:26,134 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:26,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:27,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:28,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:29,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:30,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:31,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:31,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:32,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:33,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:34,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:35,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:36,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:14:36,340 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:14:36,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:36,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:37,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:38,700 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:39,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:40,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:41,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:42,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:42,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:43,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:43,928 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:14:44,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:45,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:46,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:47,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:47,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:48,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:49,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:50,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:51,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:14:51,342 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:14:51,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:52,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:52,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:53,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:54,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:14:54,331 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:14:54,332 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:14:54,337 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:14:54,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:54,783 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:14:55,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:55,784 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:14:56,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:57,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:58,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:14:58,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:14:59,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:00,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:01,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:02,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:03,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:04,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:04,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:05,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:06,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:15:06,342 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:15:06,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:07,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:08,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:09,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:10,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:10,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:11,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:12,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:13,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:13,932 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:15:14,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:15,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:15,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:16,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:17,904 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:18,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:19,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:20,905 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:20,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:21,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:15:21,344 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:15:21,909 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:22,909 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:23,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:15:23,425 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:15:23,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:15:23,428 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:15:23,810 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:15:23,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:24,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:25,811 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:15:25,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:26,467 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:26,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:27,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:28,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:29,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:30,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:31,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:31,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:32,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:33,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:34,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:35,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:36,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:15:36,344 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:15:36,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:36,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:37,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:38,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:39,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:40,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:41,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:41,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:42,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:43,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:43,936 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:15:44,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:45,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:46,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:47,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:47,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:48,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:49,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:50,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:51,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:15:51,347 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:15:51,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:52,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:52,953 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:53,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:15:53,422 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:15:53,422 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:15:53,423 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:15:53,840 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:15:53,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:54,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:55,842 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:15:55,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:56,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:57,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:58,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:15:58,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:15:59,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:00,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:01,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:02,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:03,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:04,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:04,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:05,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:06,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:16:06,344 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:16:06,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:07,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:08,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:09,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:10,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:10,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:11,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:12,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:13,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:13,940 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:16:14,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:15,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:16,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:16,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:17,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:18,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:19,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:20,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:21,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:16:21,345 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:16:21,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:16:21,620 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:16:21,621 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:16:21,622 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:16:21,865 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:16:21,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:21,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:22,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:23,866 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:16:23,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:24,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:25,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:26,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:27,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:27,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:28,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:29,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:30,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:31,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:32,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:33,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:33,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:34,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:35,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:36,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:16:36,345 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:16:36,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:37,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:38,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:38,944 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:39,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:40,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:41,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:42,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:43,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:43,943 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:16:43,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:44,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:45,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:46,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:47,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:48,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:48,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:49,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:50,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:51,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:16:51,346 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:16:51,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:16:51,700 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:16:51,700 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:16:51,701 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:16:51,891 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:16:52,121 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:53,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:53,893 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:16:54,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:54,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:16:55,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:56,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:57,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:58,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:16:59,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:00,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:00,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:01,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:02,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:03,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:04,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:05,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:06,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:06,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:06,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:17:06,345 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:17:07,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:08,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:09,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:10,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:11,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:12,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:12,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:13,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:13,947 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:17:14,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:15,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:16,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:17,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:17,121 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:18,132 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:19,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:20,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:21,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:21,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:17:21,350 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:17:22,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:22,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:23,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:17:23,084 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:17:23,084 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:17:23,086 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:17:23,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:23,919 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:17:23,919 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:17:24,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:25,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:26,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:27,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:28,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:28,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:29,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:30,486 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:31,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:32,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:33,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:33,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:34,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:35,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:36,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:17:36,347 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:17:36,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:37,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:38,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:39,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:39,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:40,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:41,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:42,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:43,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:43,951 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:17:44,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:44,953 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:45,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:46,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:47,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:48,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:49,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:50,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:50,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:51,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:17:51,352 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:17:51,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:52,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:52,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:17:52,719 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:17:52,720 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:17:52,721 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:17:52,944 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:17:53,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:53,945 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:17:54,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:55,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:55,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:17:56,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:57,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:58,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:17:59,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:00,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:01,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:01,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:02,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:03,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:04,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:05,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:06,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:18:06,348 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:18:06,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:06,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:07,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:08,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:09,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:10,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:11,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:12,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:12,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:13,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:13,955 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:18:14,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:15,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:16,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:17,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:17,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:18,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:19,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:20,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:21,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:18:21,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:18:21,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:22,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:18:22,407 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:18:22,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:18:22,408 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:22,410 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:18:22,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:22,990 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:18:23,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:23,991 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:18:24,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:25,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:26,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:27,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:28,134 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:28,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:29,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:30,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:31,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:32,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:33,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:34,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:34,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:35,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:36,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:18:36,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:18:36,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:37,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:38,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:39,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:40,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:40,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:41,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:42,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:43,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:43,959 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:18:44,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:45,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:45,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:46,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:47,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:48,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:49,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:49,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:18:49,872 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:18:49,872 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:18:49,873 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:18:50,015 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:18:50,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:51,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:18:51,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:18:51,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:51,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:52,017 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:18:52,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:53,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:54,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:55,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:56,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:57,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:18:57,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:58,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:18:59,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:00,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:01,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:02,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:02,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:03,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:04,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:05,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:06,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:19:06,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:19:06,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:07,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:07,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:08,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:09,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:10,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:11,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:12,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:12,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:13,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:13,962 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:19:14,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:15,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:16,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:17,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:18,144 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:18,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:19,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:20,286 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:19:20,287 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:19:20,287 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:19:20,288 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:19:20,707 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:21,044 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:19:21,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:19:21,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:19:21,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:22,045 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:19:22,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:23,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:23,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:24,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:25,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:26,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:27,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:28,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:29,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:29,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:30,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:31,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:32,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:33,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:34,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:35,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:35,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:36,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:19:36,351 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:19:36,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:37,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:38,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:39,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:40,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:40,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:41,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:42,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:43,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:43,966 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:19:44,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:45,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:45,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:46,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:47,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:48,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:49,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:50,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:51,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:51,352 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:19:51,352 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:19:51,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:52,287 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:19:52,289 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:19:52,289 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:19:52,291 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:19:52,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:53,075 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:19:53,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:54,076 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:19:54,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:55,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:56,329 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:19:56,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:57,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:58,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:19:59,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:00,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:01,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:02,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:02,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:03,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:04,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:05,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:06,352 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:20:06,353 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:20:06,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:07,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:08,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:08,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:09,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:10,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:11,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:12,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:13,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:13,970 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:20:13,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:14,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:15,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:16,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:17,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:18,725 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:19,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:19,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:20,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:21,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:20:21,360 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:20:21,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:22,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:23,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:23,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:20:23,761 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:20:23,761 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:20:23,762 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:20:24,105 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:20:24,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:24,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:25,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:26,106 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:20:26,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:27,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:28,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:29,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:30,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:30,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:31,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:32,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:33,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:34,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:35,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:36,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:36,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:20:36,360 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:20:36,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:37,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:38,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:39,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:40,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:41,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:41,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:42,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:43,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:43,974 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:20:44,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:45,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:46,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:46,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:47,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:48,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:49,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:50,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:51,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:20:51,365 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:20:51,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:52,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:52,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:53,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:53,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:20:53,795 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:20:53,796 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:20:53,797 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:20:54,128 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:20:54,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:55,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:56,130 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:20:56,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:57,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:58,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:20:58,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:20:59,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:00,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:01,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:02,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:03,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:03,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:04,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:05,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:06,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:21:06,363 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:21:06,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:07,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:08,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:09,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:09,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:10,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:11,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:12,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:13,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:13,978 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:21:14,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:14,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:15,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:16,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:17,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:18,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:19,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:20,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:20,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:21,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:21:21,363 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:21:21,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:22,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:23,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:24,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:25,179 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:25,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:26,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:27,264 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:21:27,265 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:21:27,266 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:21:27,267 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:21:27,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:28,158 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:21:28,158 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:21:28,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:29,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:30,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:30,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:31,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:32,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:33,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:34,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:35,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:36,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:36,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:21:36,363 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:21:36,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:37,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:38,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:39,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:40,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:41,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:42,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:42,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:43,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:43,983 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:21:44,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:45,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:46,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:47,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:47,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:48,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:49,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:50,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:51,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:21:51,366 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:21:51,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:52,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:52,753 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:53,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:54,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:55,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:56,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:21:56,387 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:21:56,387 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:21:56,389 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:21:56,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:57,183 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:21:57,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:58,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:21:58,184 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:21:58,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:21:59,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:00,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:01,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:02,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:03,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:04,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:04,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:05,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:06,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:22:06,366 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:22:06,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:07,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:08,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:09,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:10,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:10,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:11,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:12,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:13,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:13,987 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:22:14,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:15,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:15,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:16,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:17,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:18,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:19,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:20,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:21,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:21,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:22:21,366 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:22:21,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:22,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:23,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:24,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:25,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:26,185 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:26,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:27,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:22:27,361 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:22:27,361 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:22:27,363 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:22:27,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:28,208 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:22:28,208 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:22:28,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:29,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:30,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:31,401 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:31,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:32,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:33,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:34,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:35,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:36,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:22:36,366 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:22:36,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:36,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:37,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:38,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:39,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:40,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:41,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:42,171 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:42,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:43,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:43,990 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:22:44,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:45,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:46,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:47,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:48,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:48,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:49,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:50,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:51,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:22:51,366 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:22:51,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:52,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:53,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:54,175 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:54,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:55,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:56,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:57,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:22:57,511 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:22:57,512 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:22:57,513 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:22:57,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:58,237 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:22:58,238 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:22:58,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:22:59,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:22:59,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:00,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:01,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:02,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:03,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:04,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:05,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:05,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:06,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:23:06,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:23:06,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:07,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:08,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:09,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:10,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:11,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:11,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:12,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:13,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:13,993 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:23:14,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:15,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:16,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:16,995 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:17,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:18,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:19,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:20,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:21,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:23:21,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:23:21,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:22,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:22,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:23,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:24,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:25,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:26,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:27,185 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:27,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:28,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:29,008 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:23:29,010 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:23:29,010 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:23:29,011 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:23:29,267 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:23:29,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:30,268 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:23:30,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:31,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:32,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:33,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:33,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:34,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:35,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:36,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:23:36,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:23:36,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:37,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:38,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:39,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:39,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:40,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:41,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:42,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:43,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:43,997 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:23:44,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:44,999 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:46,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:46,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:47,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:48,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:49,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:50,177 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:50,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:51,366 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:23:51,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:23:51,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:52,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:53,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:54,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:55,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:56,177 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:23:56,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:57,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:58,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:23:59,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:00,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:01,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:01,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:02,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:24:02,103 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:24:02,104 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:24:02,106 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:24:02,294 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:24:03,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:04,296 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:24:04,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:05,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:06,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:06,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:06,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:24:06,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:24:07,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:08,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:09,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:10,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:11,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:12,262 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:12,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:13,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:14,001 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:24:14,342 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:15,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:16,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:17,263 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:17,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:18,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:19,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:20,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:21,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:21,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:24:21,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:24:22,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:23,263 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:23,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:24,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:25,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:26,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:27,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:28,279 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:28,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:29,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:30,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:31,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:32,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:32,352 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:24:32,354 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:24:32,354 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:24:32,356 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:24:33,320 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:24:33,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:33,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:34,321 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:24:34,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:35,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:36,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:36,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:24:36,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:24:37,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:38,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:39,270 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:39,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:40,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:41,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:42,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:43,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:44,004 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:24:44,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:45,007 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:45,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:46,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:47,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:48,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:49,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:50,270 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:50,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:51,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:51,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:24:51,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:24:52,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:53,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:54,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:55,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:56,270 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:24:56,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:57,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:58,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:24:59,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:00,353 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:01,281 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:01,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:02,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:03,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:04,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:04,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:25:04,368 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:25:04,368 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:25:04,370 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:25:05,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:05,350 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:25:06,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:06,351 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:25:06,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:25:06,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:25:06,533 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:07,348 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:08,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:09,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:10,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:11,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:12,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:12,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:13,349 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:14,009 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:25:14,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:15,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:16,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:17,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:17,507 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:18,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:19,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:20,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:21,367 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:25:21,368 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:25:21,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:22,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:23,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:23,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:24,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:25,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:26,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:27,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:28,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:29,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:29,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:30,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:31,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:32,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:33,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:34,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:34,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:35,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:35,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:25:35,964 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:25:35,964 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:25:35,966 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:25:36,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:25:36,379 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:25:36,385 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:25:36,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:37,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:38,386 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:25:38,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:39,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:40,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:40,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:41,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:42,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:43,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:44,013 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:25:44,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:45,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:46,015 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:46,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:47,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:48,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:49,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:50,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:51,376 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:25:51,377 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:25:51,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:51,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:52,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:53,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:54,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:55,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:56,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:57,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:25:57,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:58,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:25:59,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:00,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:01,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:02,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:02,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:03,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:04,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:05,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:05,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:26:05,586 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:26:05,587 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:26:05,588 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:26:06,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:26:06,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:26:06,412 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:26:06,412 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:26:06,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:07,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:07,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:08,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:09,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:10,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:11,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:12,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:13,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:13,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:14,017 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:26:14,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:15,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:16,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:17,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:18,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:19,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:19,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:20,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:21,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:26:21,384 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:26:21,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:22,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:23,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:24,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:24,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:25,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:26,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:27,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:28,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:29,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:30,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:30,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:31,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:32,429 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:33,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:34,436 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:35,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:35,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:36,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:26:36,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:26:36,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:37,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:38,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:38,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:26:38,511 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:26:38,512 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:26:38,513 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:26:39,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:39,438 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:26:40,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:40,439 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:26:40,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:41,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:42,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:43,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:44,020 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:26:44,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:45,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:46,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:46,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:47,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:48,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:49,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:50,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:51,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:26:51,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:26:51,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:51,541 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:52,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:53,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:54,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:55,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:56,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:56,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:26:57,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:58,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:26:59,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:00,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:01,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:02,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:02,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:03,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:04,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:05,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:06,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:27:06,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:27:06,649 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:07,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:07,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:08,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:09,652 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:10,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:11,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:27:11,379 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:27:11,380 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:27:11,381 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:27:11,475 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:27:11,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:12,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:13,476 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:27:13,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:13,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:14,024 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:27:14,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:15,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:16,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:17,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:18,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:19,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:19,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:20,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:21,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:27:21,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:27:21,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:22,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:23,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:24,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:25,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:25,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:26,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:27,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:28,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:29,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:30,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:30,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:31,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:32,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:33,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:34,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:35,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:35,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:36,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:27:36,387 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:27:36,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:37,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:38,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:39,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:40,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:40,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:41,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:27:41,061 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:27:41,061 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:27:41,063 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:27:41,510 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:27:41,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:42,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:43,511 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:27:43,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:44,027 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:27:44,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:45,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:46,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:46,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:47,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:48,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:49,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:50,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:51,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:27:51,387 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:27:51,507 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:51,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:52,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:53,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:54,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:55,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:56,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:27:56,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:57,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:58,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:27:59,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:00,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:01,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:01,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:02,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:03,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:04,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:05,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:06,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:28:06,400 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:28:06,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:07,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:07,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:08,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:09,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:10,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:10,997 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:28:10,999 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:28:10,999 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:28:11,001 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:28:11,540 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:28:11,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:12,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:12,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:13,541 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:28:13,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:14,031 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:28:14,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:15,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:16,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:17,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:18,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:18,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:19,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:20,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:21,389 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:28:21,390 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:28:21,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:22,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:23,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:24,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:24,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:25,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:26,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:27,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:28,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:29,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:29,802 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:30,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:31,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:32,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:33,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:34,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:34,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:35,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:36,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:28:36,405 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:28:36,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:37,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:38,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:39,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:39,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:40,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:41,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:28:41,624 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:28:41,625 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:28:41,626 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:28:42,044 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:42,565 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:28:43,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:43,566 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:28:44,035 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:28:44,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:45,038 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:45,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:46,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:47,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:48,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:49,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:50,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:50,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:51,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:51,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:28:51,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:28:52,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:53,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:54,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:55,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:56,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:56,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:28:57,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:58,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:28:59,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:00,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:01,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:01,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:02,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:03,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:04,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:05,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:06,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:06,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:29:06,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:29:07,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:07,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:08,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:09,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:10,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:11,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:12,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:12,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:13,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:13,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:29:13,499 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:29:13,499 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:29:13,500 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:29:13,596 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:29:14,040 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:29:14,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:15,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:15,598 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:29:16,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:17,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:17,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:18,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:19,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:20,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:21,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:21,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:29:21,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:29:22,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:23,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:23,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:24,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:25,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:26,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:27,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:28,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:28,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:29,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:30,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:31,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:32,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:33,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:34,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:34,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:35,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:36,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:36,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:29:36,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:29:37,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:38,077 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:39,081 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:39,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:40,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:41,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:42,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:43,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:29:43,047 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:29:43,047 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:29:43,049 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:29:43,460 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:43,624 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:29:44,044 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:29:44,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:45,047 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:45,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:45,626 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:29:46,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:47,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:48,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:49,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:50,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:50,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:51,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:51,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:29:51,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:29:52,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:53,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:54,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:55,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:56,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:56,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:29:57,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:58,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:29:59,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:00,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:01,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:01,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:02,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:03,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:04,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:05,201 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:06,198 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:06,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:30:06,391 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:30:07,205 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:07,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:08,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:09,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:10,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:10,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:30:10,772 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:30:10,772 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:30:10,774 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:30:11,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:11,651 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:30:12,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:12,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:13,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:13,652 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:30:14,048 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:30:14,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:15,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:16,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:17,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:18,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:18,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:19,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:20,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:21,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:21,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:30:21,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:30:22,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:23,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:24,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:24,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:25,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:26,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:27,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:28,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:29,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:29,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:30,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:31,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:32,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:33,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:34,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:35,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:35,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:36,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:36,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:30:36,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:30:37,215 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:38,212 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:39,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:40,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:40,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:41,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:42,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:42,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:30:42,644 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:30:42,645 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:30:42,646 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:30:42,673 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:30:43,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:43,674 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:30:44,052 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:30:44,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:45,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:46,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:46,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:47,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:48,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:49,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:50,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:51,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:51,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:30:51,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:30:51,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:52,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:53,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:54,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:55,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:56,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:56,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:30:57,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:58,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:30:59,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:00,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:01,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:02,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:02,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:03,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:04,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:05,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:06,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:06,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:31:06,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:31:07,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:08,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:08,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:09,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:10,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:11,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:12,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:13,227 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:14,056 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:31:14,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:14,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:31:14,205 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:31:14,205 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:31:14,207 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:31:14,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:14,706 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:31:15,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:15,710 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:31:16,353 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:17,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:18,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:19,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:19,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:20,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:21,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:21,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:31:21,392 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:31:22,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:23,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:24,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:25,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:25,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:26,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:27,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:28,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:29,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:30,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:30,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:31,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:32,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:33,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:34,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:35,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:36,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:36,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:31:36,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:31:36,564 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:37,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:38,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:39,361 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:40,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:41,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:41,496 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:31:41,497 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:31:41,497 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:31:41,499 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:31:41,743 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:31:42,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:42,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:43,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:43,744 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:31:44,060 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:31:44,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:45,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:46,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:47,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:47,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:48,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:49,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:50,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:51,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:51,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:31:51,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:31:52,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:53,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:53,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:54,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:55,363 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:56,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:57,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:58,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:31:58,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:31:59,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:00,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:01,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:02,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:03,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:04,364 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:04,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:05,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:06,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:06,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:32:06,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:32:07,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:08,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:09,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:09,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:10,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:11,375 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:12,375 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:13,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:13,889 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:32:13,891 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:32:13,891 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:32:13,893 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:32:14,064 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:32:14,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:14,776 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:32:15,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:15,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:15,778 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:32:16,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:17,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:18,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:19,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:20,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:20,587 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:21,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:21,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:32:21,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:32:22,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:23,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:24,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:25,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:26,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:26,587 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:27,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:28,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:29,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:30,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:31,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:31,587 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:32,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:33,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:34,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:35,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:36,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:36,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:32:36,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:32:37,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:37,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:38,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:39,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:40,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:41,382 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:42,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:42,593 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:43,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:44,069 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:32:44,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:45,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:45,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:32:45,975 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:32:45,975 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:32:45,976 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:32:46,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:46,803 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:32:47,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:47,804 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:32:48,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:48,591 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:49,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:50,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:51,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:51,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:32:51,397 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:32:52,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:53,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:54,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:54,591 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:32:55,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:56,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:57,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:58,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:32:59,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:00,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:00,592 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:01,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:02,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:03,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:04,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:05,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:05,592 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:06,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:06,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:33:06,397 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:33:07,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:08,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:09,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:10,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:11,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:11,592 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:12,388 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:13,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:14,078 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:33:14,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:15,391 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:16,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:17,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:17,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:18,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:18,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:33:18,498 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:33:18,499 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:33:18,500 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:33:18,834 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:33:19,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:19,835 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:33:20,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:21,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:21,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:33:21,397 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:33:22,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:22,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:23,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:24,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:25,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:26,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:27,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:27,594 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:28,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:29,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:30,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:31,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:32,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:33,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:33,594 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:34,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:35,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:36,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:33:36,397 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:33:36,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:37,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:38,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:39,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:39,595 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:40,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:41,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:42,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:43,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:44,076 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:33:44,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:45,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:45,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:46,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:47,556 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:48,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:49,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:50,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:50,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:51,401 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:33:51,401 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:33:51,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:52,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:53,037 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:33:53,038 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:33:53,038 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:33:53,043 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:33:53,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:53,862 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:33:54,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:55,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:55,864 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:33:56,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:33:56,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:57,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:58,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:33:59,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:00,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:01,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:01,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:02,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:03,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:04,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:05,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:06,401 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:34:06,401 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:34:06,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:07,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:07,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:08,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:09,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:10,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:11,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:12,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:12,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:13,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:14,080 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:34:14,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:15,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:16,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:17,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:18,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:18,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:19,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:20,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:21,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:34:21,404 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:34:21,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:22,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:23,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:23,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:24,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:24,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:34:24,571 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:34:24,571 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:34:24,574 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:34:24,890 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:34:25,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:25,891 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:34:26,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:27,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:28,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:29,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:29,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:30,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:31,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:32,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:33,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:34,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:35,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:35,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:36,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:34:36,403 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:34:36,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:37,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:38,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:39,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:40,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:40,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:41,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:42,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:43,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:44,084 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:34:44,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:45,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:46,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:46,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:47,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:48,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:49,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:50,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:51,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:34:51,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:34:51,518 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:51,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:52,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:53,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:54,563 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:55,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:56,567 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:56,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:34:56,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:34:56,906 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:34:56,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:34:56,912 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:34:56,915 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:34:57,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:57,917 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:34:58,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:34:59,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:00,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:01,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:02,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:02,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:03,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:04,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:05,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:06,408 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:35:06,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:35:06,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:07,568 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:08,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:08,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:09,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:10,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:11,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:12,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:13,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:13,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:14,088 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:35:14,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:15,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:16,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:17,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:18,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:19,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:19,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:20,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:21,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:35:21,420 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:35:21,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:22,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:23,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:24,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:24,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:25,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:26,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:35:26,005 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:35:26,006 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:35:26,007 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:35:26,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:26,941 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:35:27,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:27,942 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:35:28,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:29,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:30,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:30,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:31,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:32,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:33,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:34,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:35,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:36,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:35:36,414 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:35:36,526 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:36,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:37,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:38,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:39,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:40,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:41,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:41,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:42,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:43,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:44,092 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:35:44,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:45,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:46,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:47,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:47,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:48,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:49,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:50,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:51,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:35:51,414 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:35:51,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:52,533 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:52,579 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:53,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:54,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:55,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:56,583 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:57,584 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:57,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:35:58,584 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:58,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:35:58,963 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:35:58,964 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:35:58,969 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:35:58,970 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:35:59,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:35:59,970 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:36:00,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:01,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:02,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:03,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:03,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:04,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:05,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:06,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:36:06,414 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:36:06,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:07,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:08,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:09,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:09,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:10,586 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:11,808 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:12,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:13,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:14,096 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:36:14,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:15,098 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:15,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:16,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:17,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:18,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:19,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:20,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:20,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:21,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:36:21,414 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:36:21,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:22,808 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:23,808 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:24,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:25,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:25,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:26,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:27,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:28,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:29,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:29,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:36:29,932 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:36:29,932 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:36:29,937 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:36:29,998 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:36:30,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:30,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:31,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:31,999 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:36:32,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:33,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:34,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:35,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:36,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:36:36,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:36:36,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:36,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:37,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:38,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:39,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:40,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:41,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:41,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:42,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:43,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:44,100 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:36:44,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:45,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:46,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:47,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:47,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:48,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:49,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:50,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:51,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:36:51,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:36:51,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:52,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:52,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:53,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:54,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:55,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:56,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:57,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:36:57,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:58,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:36:59,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:00,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:37:00,164 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:37:00,164 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:37:00,168 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:37:00,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:01,026 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:37:01,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:02,027 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:37:02,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:02,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:03,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:04,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:05,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:06,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:37:06,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:37:06,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:07,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:08,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:08,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:09,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:10,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:11,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:12,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:13,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:13,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:14,104 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:37:14,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:15,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:16,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:17,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:18,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:18,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:19,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:20,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:21,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:37:21,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:37:21,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:22,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:23,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:24,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:24,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:25,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:26,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:27,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:28,841 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:29,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:29,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:30,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:31,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:37:31,608 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:37:31,609 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:37:31,614 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:37:32,055 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:37:32,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:33,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:34,056 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:37:34,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:34,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:35,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:36,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:37:36,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:37:36,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:37,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:38,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:39,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:40,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:40,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:41,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:42,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:43,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:44,109 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:37:44,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:45,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:46,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:46,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:47,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:48,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:49,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:50,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:51,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:37:51,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:37:51,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:51,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:52,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:53,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:54,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:55,841 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:56,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:37:56,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:57,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:58,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:37:59,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:00,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:01,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:38:01,415 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:38:01,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:38:01,417 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:38:01,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:02,087 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:38:02,456 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:02,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:03,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:04,089 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:38:04,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:05,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:06,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:38:06,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:38:06,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:07,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:07,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:08,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:09,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:10,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:11,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:12,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:12,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:13,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:14,111 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:38:14,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:15,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:16,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:17,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:17,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:18,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:19,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:20,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:21,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:38:21,415 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:38:21,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:22,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:23,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:23,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:24,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:25,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:26,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:27,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:28,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:28,966 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:29,966 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:30,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:31,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:32,444 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:38:32,445 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:38:32,446 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:38:32,447 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:38:33,116 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:38:33,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:34,117 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:38:34,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:34,573 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:35,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:36,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:38:36,417 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:38:36,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:37,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:38,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:39,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:40,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:40,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:41,570 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:42,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:43,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:44,115 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:38:44,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:45,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:46,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:46,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:47,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:48,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:49,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:50,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:51,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:38:51,416 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:38:51,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:51,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:52,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:53,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:54,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:55,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:56,571 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:38:56,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:57,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:58,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:38:59,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:00,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:01,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:01,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:02,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:03,580 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:04,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:39:04,205 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:39:04,206 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:39:04,208 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:39:04,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:05,147 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:39:05,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:06,148 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:39:06,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:39:06,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:39:06,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:07,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:07,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:08,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:09,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:10,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:11,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:12,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:12,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:13,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:14,119 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:39:14,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:15,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:16,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:17,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:18,121 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:18,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:19,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:20,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:21,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:39:21,417 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:39:21,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:22,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:23,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:23,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:24,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:25,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:26,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:27,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:28,575 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:28,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:29,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:30,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:31,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:32,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:33,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:33,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:34,484 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:39:34,485 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:39:34,486 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:39:34,489 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:39:34,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:35,172 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:39:35,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:36,173 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:39:36,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:39:36,417 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:39:36,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:37,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:38,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:39,576 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:39,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:40,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:41,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:42,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:43,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:44,123 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:39:44,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:45,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:45,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:46,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:47,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:48,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:49,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:50,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:50,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:51,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:39:51,417 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:39:51,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:52,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:53,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:54,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:55,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:56,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:39:56,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:57,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:58,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:39:59,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:00,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:01,593 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:01,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:02,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:03,645 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:04,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:40:04,004 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:40:04,005 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:40:04,007 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:40:04,201 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:40:04,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:05,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:06,203 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:40:06,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:40:06,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:40:06,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:07,521 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:07,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:08,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:09,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:10,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:11,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:12,578 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:12,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:13,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:14,127 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:40:14,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:15,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:16,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:17,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:18,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:18,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:19,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:20,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:21,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:40:21,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:40:21,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:22,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:23,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:23,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:24,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:25,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:26,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:27,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:28,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:28,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:29,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:30,657 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:31,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:32,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:33,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:33,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:40:33,628 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:40:33,629 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:40:33,631 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:40:34,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:34,234 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:40:34,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:35,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:36,235 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:40:36,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:40:36,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:40:36,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:37,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:38,581 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:38,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:39,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:40,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:41,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:42,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:43,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:44,131 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:40:44,134 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:44,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:45,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:46,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:47,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:48,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:49,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:49,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:50,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:51,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:40:51,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:40:51,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:52,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:53,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:54,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:55,582 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:40:55,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:56,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:57,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:58,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:40:59,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:00,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:00,780 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:01,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:02,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:03,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:04,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:05,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:05,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:41:05,867 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:41:05,868 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:41:05,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:05,871 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:41:06,261 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:41:06,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:41:06,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:41:06,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:07,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:08,262 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:41:08,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:09,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:10,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:11,584 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:11,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:12,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:13,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:14,135 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:41:14,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:15,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:16,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:17,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:17,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:18,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:19,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:20,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:21,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:41:21,418 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:41:21,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:22,526 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:22,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:23,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:24,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:25,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:26,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:27,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:27,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:28,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:29,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:30,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:31,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:32,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:32,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:33,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:34,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:35,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:36,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:41:36,424 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:41:36,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:37,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:37,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:37,888 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:41:37,890 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:41:37,890 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:41:37,896 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:41:38,294 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:41:38,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:39,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:40,295 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:41:40,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:41,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:42,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:43,588 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:43,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:44,139 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:41:44,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:45,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:46,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:47,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:48,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:49,588 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:49,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:50,909 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:51,420 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:41:51,421 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:41:51,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:52,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:53,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:54,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:54,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:55,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:56,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:57,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:58,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:41:59,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:41:59,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:00,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:01,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:02,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:03,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:04,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:04,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:05,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:06,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:42:06,430 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:42:06,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:07,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:08,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:42:08,718 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:42:08,719 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:42:08,722 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:42:09,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:09,317 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:42:09,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:09,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:10,318 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:42:10,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:11,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:12,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:13,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:14,143 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:42:14,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:15,145 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:15,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:16,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:17,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:18,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:19,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:20,595 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:20,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:21,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:42:21,425 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:42:21,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:22,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:23,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:24,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:25,595 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:25,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:26,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:27,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:28,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:29,873 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:30,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:31,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:31,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:32,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:33,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:34,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:35,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:36,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:42:36,425 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:42:36,673 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:42:36,674 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:42:36,674 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:42:36,675 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:36,676 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:42:37,096 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:37,343 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:42:38,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:38,344 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:42:39,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:40,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:41,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:42,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:42,599 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:43,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:44,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:44,147 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:42:45,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:46,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:47,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:48,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:48,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:49,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:50,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:51,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:51,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:42:51,425 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:42:52,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:53,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:53,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:54,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:55,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:56,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:57,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:58,104 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:42:58,599 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:42:59,098 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:00,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:01,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:02,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:03,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:03,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:43:03,236 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:43:03,237 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:43:03,239 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:43:03,369 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:43:03,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:04,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:04,370 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:43:05,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:06,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:06,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:43:06,425 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:43:07,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:08,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:09,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:09,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:10,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:11,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:12,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:13,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:14,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:14,151 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:43:15,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:15,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:16,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:17,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:18,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:19,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:20,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:20,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:21,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:21,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:43:21,425 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:43:22,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:23,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:24,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:25,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:25,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:26,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:27,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:28,109 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:29,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:30,109 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:30,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:31,109 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:32,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:33,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:33,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:43:33,734 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:43:33,734 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:43:33,735 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:43:34,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:34,402 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:43:34,402 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:43:35,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:36,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:36,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:36,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:43:36,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:43:37,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:38,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:39,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:40,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:41,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:42,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:42,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:43,149 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:44,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:44,155 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:43:45,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:46,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:47,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:47,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:48,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:49,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:50,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:51,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:51,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:43:51,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:43:52,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:52,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:53,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:54,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:55,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:56,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:57,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:58,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:43:58,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:43:59,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:00,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:01,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:02,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:02,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:44:02,531 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:44:02,531 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:44:02,532 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:44:03,434 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:44:03,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:03,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:04,435 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:44:04,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:05,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:06,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:44:06,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:44:06,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:07,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:08,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:09,150 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:09,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:10,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:11,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:12,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:13,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:14,159 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:44:14,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:14,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:15,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:16,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:17,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:18,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:19,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:20,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:20,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:21,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:44:21,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:44:21,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:22,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:23,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:24,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:25,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:26,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:26,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:27,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:28,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:29,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:30,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:31,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:31,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:32,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:44:32,394 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:44:32,395 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:44:32,396 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:44:32,464 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:44:33,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:34,466 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:44:34,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:35,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:36,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:44:36,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:44:36,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:36,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:37,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:38,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:39,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:40,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:41,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:42,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:42,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:43,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:44,163 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:44:44,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:45,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:46,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:47,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:47,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:48,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:49,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:50,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:51,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:44:51,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:44:51,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:52,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:52,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:53,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:54,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:55,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:56,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:57,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:58,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:44:58,622 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:44:59,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:00,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:01,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:02,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:45:02,424 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:45:02,424 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:45:02,425 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:45:02,495 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:45:03,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:03,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:04,497 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:45:04,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:05,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:06,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:45:06,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:45:06,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:07,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:08,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:08,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:09,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:10,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:11,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:12,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:13,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:14,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:14,167 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:45:14,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:15,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:16,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:17,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:18,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:19,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:19,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:20,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:21,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:45:21,427 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:45:21,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:22,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:23,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:24,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:25,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:25,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:26,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:27,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:28,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:29,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:30,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:30,526 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:31,526 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:32,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:33,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:34,539 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:35,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:45:35,241 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:45:35,242 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:45:35,242 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:35,243 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:45:35,522 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:45:35,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:36,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:45:36,427 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:45:36,523 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:45:36,694 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:37,694 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:38,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:39,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:40,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:41,154 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:41,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:42,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:43,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:44,171 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:45:44,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:45,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:46,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:46,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:47,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:48,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:49,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:50,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:51,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:45:51,426 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:45:51,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:51,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:52,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:53,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:54,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:55,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:56,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:57,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:45:57,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:58,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:45:59,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:00,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:01,702 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:02,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:02,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:03,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:04,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:05,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:06,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:46:06,427 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:46:06,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:06,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:46:06,746 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:46:06,746 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:46:06,748 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:46:07,556 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:46:07,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:07,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:08,557 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:46:08,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:09,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:10,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:11,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:12,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:13,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:13,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:14,175 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:46:14,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:15,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:16,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:17,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:18,178 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:18,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:19,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:20,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:21,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:46:21,427 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:46:21,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:22,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:23,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:23,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:24,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:25,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:26,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:27,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:28,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:29,166 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:29,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:30,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:31,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:32,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:33,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:34,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:34,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:35,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:36,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:46:36,434 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:46:36,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:37,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:38,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:46:38,056 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:46:38,056 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:46:38,058 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:46:38,586 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:46:38,587 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:46:38,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:39,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:39,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:40,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:41,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:42,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:43,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:44,179 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:46:44,185 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:44,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:45,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:46,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:47,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:48,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:49,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:50,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:50,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:51,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:46:51,434 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:46:51,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:52,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:53,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:54,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:55,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:56,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:46:56,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:57,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:58,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:46:59,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:00,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:01,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:02,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:02,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:03,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:04,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:05,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:06,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:47:06,440 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:47:06,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:07,529 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:07,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:08,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:09,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:10,586 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:47:10,587 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:47:10,587 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:47:10,589 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:47:10,612 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:47:11,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:12,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:12,614 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:47:12,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:13,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:14,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:14,183 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:47:15,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:16,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:17,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:18,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:18,185 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:19,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:20,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:21,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:21,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:47:21,440 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:47:22,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:23,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:23,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:24,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:25,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:26,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:27,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:28,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:29,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:29,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:30,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:31,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:32,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:33,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:34,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:35,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:35,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:36,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:36,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:47:36,441 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:47:37,115 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:38,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:39,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:40,121 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:40,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:41,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:41,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:47:41,387 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:47:41,388 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:47:41,389 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:47:41,640 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:47:42,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:42,641 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:47:43,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:44,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:44,187 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:47:45,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:45,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:46,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:47,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:48,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:49,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:50,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:51,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:51,176 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:51,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:47:51,440 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:47:52,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:53,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:54,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:55,186 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:56,186 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:56,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:47:57,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:58,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:47:59,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:00,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:01,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:02,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:02,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:03,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:04,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:05,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:06,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:06,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:48:06,447 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:48:07,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:07,590 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:08,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:09,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:10,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:11,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:11,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:48:11,551 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:48:11,551 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:48:11,552 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:48:11,662 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:48:12,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:12,591 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:12,663 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:48:13,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:14,191 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:48:14,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:15,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:16,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:17,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:18,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:18,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:19,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:20,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:21,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:21,444 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:48:21,444 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:48:22,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:23,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:23,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:24,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:25,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:26,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:27,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:28,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:29,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:29,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:30,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:31,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:32,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:33,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:34,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:34,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:35,201 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:36,205 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:36,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:48:36,448 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:48:37,205 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:38,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:39,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:39,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:48:39,822 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:48:39,822 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:48:39,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:39,823 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:48:40,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:40,686 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:48:40,686 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:48:41,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:42,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:43,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:44,194 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:48:44,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:45,201 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:45,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:46,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:47,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:48,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:49,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:50,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:51,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:51,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:51,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:48:51,448 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:48:52,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:53,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:54,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:55,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:56,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:48:56,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:57,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:58,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:48:59,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:00,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:01,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:02,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:02,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:03,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:04,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:05,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:06,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:06,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:49:06,459 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:49:07,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:07,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:08,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:09,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:10,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:11,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:12,038 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:49:12,040 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:49:12,040 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:49:12,041 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:49:12,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:12,713 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:49:12,713 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:49:13,080 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:13,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:14,198 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:49:14,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:15,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:16,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:17,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:18,205 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:18,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:19,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:20,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:21,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:49:21,451 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:49:21,491 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:22,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:23,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:23,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:24,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:25,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:26,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:27,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:28,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:29,200 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:29,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:30,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:31,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:32,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:33,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:34,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:35,200 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:35,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:36,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:49:36,451 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:49:36,499 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:37,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:38,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:39,467 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:40,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:40,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:41,292 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:49:41,294 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:49:41,294 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:49:41,296 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:49:41,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:41,741 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:49:42,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:42,742 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:49:43,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:44,202 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:49:44,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:45,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:45,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:46,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:47,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:48,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:49,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:50,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:51,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:51,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:49:51,451 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:49:51,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:52,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:53,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:54,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:55,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:56,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:49:56,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:57,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:58,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:49:59,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:00,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:01,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:02,203 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:02,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:03,719 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:04,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:05,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:06,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:50:06,454 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:50:06,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:07,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:07,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:08,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:50:08,718 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:50:08,718 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:50:08,720 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:50:08,758 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:08,768 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:50:09,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:10,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:10,771 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:50:11,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:12,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:13,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:13,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:14,206 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:50:14,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:15,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:16,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:17,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:18,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:18,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:19,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:20,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:21,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:50:21,454 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:50:21,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:22,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:23,589 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:23,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:24,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:25,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:26,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:27,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:28,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:29,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:29,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:30,722 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:31,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:32,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:33,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:34,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:35,212 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:35,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:36,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:50:36,457 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:50:36,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:37,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:38,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:39,674 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:50:39,675 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:50:39,675 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:50:39,677 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:50:39,796 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:50:40,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:40,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:40,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:41,798 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:50:41,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:42,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:43,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:44,210 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:50:44,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:45,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:46,212 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:46,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:47,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:48,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:49,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:50,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:51,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:50:51,458 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:50:51,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:51,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:52,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:53,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:54,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:55,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:56,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:57,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:50:57,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:58,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:50:59,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:00,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:01,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:02,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:02,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:03,836 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:04,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:05,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:06,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:51:06,468 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:51:06,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:07,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:07,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:08,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:09,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:51:09,384 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:51:09,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:51:09,386 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:51:09,822 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:51:09,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:10,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:11,823 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:51:11,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:12,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:13,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:13,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:14,214 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:51:14,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:15,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:16,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:17,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:18,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:19,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:19,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:20,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:21,461 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:51:21,462 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:51:21,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:22,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:23,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:24,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:24,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:25,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:26,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:27,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:28,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:29,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:29,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:30,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:31,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:32,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:33,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:34,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:34,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:35,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:36,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:51:36,462 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:51:36,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:37,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:38,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:39,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:39,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:51:39,439 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:51:39,439 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:51:39,440 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:51:39,846 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:51:39,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:40,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:41,848 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:51:41,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:42,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:43,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:44,219 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:51:44,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:44,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:45,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:46,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:47,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:48,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:49,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:50,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:50,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:51,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:51:51,462 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:51:51,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:52,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:53,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:54,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:55,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:51:55,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:56,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:57,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:58,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:51:59,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:00,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:00,867 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:01,867 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:02,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:03,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:04,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:05,300 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:52:05,301 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:52:05,302 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:52:05,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:05,304 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:52:05,865 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:52:05,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:06,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:52:06,462 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:52:06,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:07,866 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:52:07,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:08,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:09,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:10,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:11,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:11,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:12,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:13,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:14,222 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:52:14,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:15,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:16,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:16,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:17,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:18,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:19,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:20,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:21,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:52:21,462 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:52:21,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:21,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:22,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:23,869 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:24,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:25,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:26,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:27,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:27,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:28,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:29,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:30,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:31,870 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:32,873 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:33,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:33,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:34,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:35,874 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:36,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:52:36,477 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:52:36,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:37,875 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:38,488 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:52:38,488 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:52:38,489 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:52:38,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:38,490 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:52:38,901 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:52:38,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:39,902 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:52:39,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:40,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:41,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:42,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:43,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:44,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:44,227 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:52:44,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:45,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:46,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:47,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:48,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:49,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:50,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:50,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:51,468 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:52:51,469 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:52:51,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:52,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:53,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:54,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:55,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:56,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:52:56,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:57,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:58,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:52:59,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:00,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:01,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:01,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:02,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:03,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:04,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:05,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:06,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:06,476 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:53:06,483 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:53:06,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:07,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:08,227 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:53:08,228 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:53:08,229 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:53:08,230 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:53:08,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:08,936 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:53:09,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:09,937 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:53:10,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:11,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:12,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:12,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:13,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:14,230 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:53:14,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:15,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:16,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:17,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:17,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:18,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:19,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:20,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:21,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:53:21,474 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:53:21,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:22,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:22,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:23,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:24,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:25,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:26,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:27,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:28,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:28,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:29,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:30,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:31,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:32,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:33,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:33,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:34,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:35,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:36,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:53:36,085 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:53:36,086 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:53:36,087 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:53:36,500 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:53:36,500 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:53:36,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:36,963 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:53:37,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:37,964 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:53:38,572 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:38,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:39,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:40,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:41,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:42,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:43,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:44,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:44,235 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:53:44,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:45,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:46,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:47,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:48,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:49,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:50,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:50,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:51,499 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:53:51,499 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:53:51,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:52,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:53,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:54,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:55,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:56,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:53:56,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:57,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:58,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:53:59,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:00,939 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:01,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:01,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:02,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:03,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:04,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:05,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:06,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:06,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:54:06,371 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:54:06,372 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:54:06,373 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:54:06,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:54:06,784 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:54:06,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:06,984 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:54:07,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:07,985 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:54:08,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:09,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:10,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:11,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:12,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:12,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:13,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:14,239 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:54:14,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:15,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:16,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:17,242 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:17,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:18,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:19,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:20,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:21,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:54:21,518 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:54:21,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:22,654 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:22,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:23,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:24,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:25,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:26,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:27,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:28,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:28,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:29,938 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:30,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:31,941 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:32,942 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:33,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:33,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:34,947 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:35,945 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:36,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:54:36,518 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:54:36,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:37,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:54:37,562 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:54:37,562 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:54:37,563 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:54:37,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:38,010 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:54:38,602 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:38,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:39,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:40,012 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:54:40,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:41,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:42,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:43,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:44,242 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:44,243 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:54:44,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:45,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:46,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:47,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:48,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:49,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:50,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:50,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:51,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:54:51,518 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:54:51,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:52,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:53,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:54,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:55,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:54:55,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:56,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:57,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:58,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:54:59,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:00,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:01,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:01,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:02,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:03,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:04,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:05,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:06,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:06,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:55:06,520 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:55:06,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:07,499 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:55:07,500 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:55:07,500 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:55:07,502 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:55:07,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:08,034 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:55:08,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:09,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:10,036 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:55:10,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:11,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:11,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:12,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:13,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:14,247 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:55:14,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:15,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:16,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:16,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:17,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:18,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:19,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:20,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:21,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:55:21,520 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:55:21,678 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:21,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:22,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:23,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:24,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:25,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:26,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:27,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:27,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:28,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:29,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:30,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:31,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:32,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:33,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:33,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:34,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:35,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:36,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:55:36,526 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:55:36,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:37,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:38,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:55:38,484 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:55:38,485 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:55:38,485 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:38,486 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:55:38,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:39,062 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:55:39,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:40,063 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:55:40,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:41,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:42,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:43,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:44,250 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:55:44,257 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:44,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:45,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:46,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:47,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:48,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:49,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:50,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:50,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:51,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:55:51,524 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:55:51,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:52,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:53,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:54,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:55,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:55:55,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:56,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:57,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:58,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:55:59,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:00,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:01,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:01,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:02,991 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:03,998 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:04,997 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:05,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:06,270 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:06,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:56:06,526 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:56:06,997 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:07,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:08,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:56:08,985 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:56:08,986 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:56:08,987 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:56:09,090 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:56:09,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:10,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:11,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:11,398 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:12,093 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:56:12,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:13,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:14,254 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:56:14,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:15,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:16,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:17,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:17,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:18,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:19,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:20,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:21,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:21,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:56:21,526 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:56:22,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:22,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:23,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:24,518 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:25,518 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:26,518 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:27,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:28,395 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:28,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:29,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:30,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:31,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:32,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:33,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:34,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:34,521 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:35,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:36,524 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:36,530 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:56:36,531 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:56:37,528 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:38,525 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:39,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:56:39,515 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:56:39,516 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:56:39,516 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:39,517 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:56:39,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:40,129 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:56:40,660 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:41,660 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:42,130 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:56:42,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:43,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:44,259 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:56:44,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:45,398 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:45,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:46,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:47,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:48,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:49,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:50,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:51,398 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:51,531 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:56:51,531 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:56:51,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:52,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:53,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:54,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:55,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:56,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:56:56,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:57,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:58,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:56:59,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:00,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:01,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:02,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:02,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:03,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:04,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:05,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:06,542 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:57:06,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:57:06,672 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:07,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:07,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:08,673 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:09,675 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:10,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:11,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:12,354 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:57:12,356 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:57:12,356 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:57:12,357 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:57:12,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:13,159 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:57:13,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:13,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:14,160 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:57:14,262 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:57:14,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:15,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:16,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:17,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:18,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:18,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:19,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:20,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:21,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:57:21,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:57:21,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:22,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:23,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:23,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:24,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:25,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:26,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:27,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:28,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:29,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:29,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:30,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:31,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:32,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:33,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:34,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:35,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:35,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:36,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:57:36,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:57:36,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:37,778 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:38,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:39,781 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:40,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:40,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:41,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:42,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:42,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:57:42,896 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:57:42,897 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:57:42,898 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:57:43,185 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:57:43,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:44,186 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:57:44,266 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:57:44,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:45,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:45,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:46,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:47,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:48,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:49,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:50,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:50,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:51,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:57:51,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:57:51,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:52,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:53,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:54,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:55,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:57:55,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:56,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:57,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:58,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:57:59,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:00,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:01,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:01,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:02,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:03,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:04,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:05,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:06,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:58:06,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:58:06,664 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:06,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:07,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:08,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:09,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:10,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:11,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:12,416 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:12,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:13,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:14,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:58:14,156 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:58:14,157 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:58:14,159 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:58:14,212 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:58:14,269 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:58:14,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:15,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:16,213 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:58:16,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:17,602 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:17,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:18,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:19,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:20,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:21,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:58:21,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:58:21,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:22,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:22,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:23,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:24,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:25,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:26,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:27,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:28,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:28,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:29,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:30,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:31,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:32,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:33,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:34,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:34,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:35,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:36,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:58:36,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:58:36,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:37,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:38,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:39,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:39,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:40,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:41,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:42,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:43,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:44,272 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:58:44,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:44,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:58:44,820 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:58:44,821 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:58:44,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:44,822 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:58:45,242 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:58:45,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:46,243 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:58:46,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:47,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:48,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:49,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:50,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:50,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:51,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:58:51,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:58:51,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:52,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:53,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:54,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:55,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:56,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:58:56,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:57,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:58,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:58:59,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:00,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:01,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:01,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:02,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:03,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:04,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:05,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:06,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:59:06,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:59:06,657 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:06,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:07,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:08,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:09,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:10,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:11,814 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:12,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:12,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:13,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:14,276 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:59:14,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:15,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:59:15,348 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:59:15,348 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:59:15,350 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:59:15,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:16,276 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:59:16,276 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:59:16,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:17,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:18,389 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:18,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:19,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:20,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:21,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:59:21,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:59:21,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:22,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:23,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:23,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:24,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:25,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:26,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:27,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:28,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:29,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:29,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:30,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:31,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:32,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:33,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:34,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:35,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:35,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:36,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:59:36,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:59:36,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:37,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:38,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:39,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:40,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:40,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:41,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:42,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:43,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:44,282 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 09:59:44,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:45,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 09:59:45,003 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 09:59:45,003 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 09:59:45,005 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 09:59:45,298 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 09:59:45,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:46,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:46,299 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 09:59:46,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:47,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:48,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:49,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:50,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:51,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 09:59:51,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 09:59:51,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:51,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:52,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:53,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:54,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:55,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:56,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:57,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 09:59:57,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:58,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 09:59:59,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:00,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:01,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:02,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:02,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:03,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:04,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:05,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:06,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:00:06,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:00:06,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:07,666 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:07,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:08,819 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:09,820 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:10,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:11,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:12,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:13,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:13,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:14,283 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:00:14,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:15,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:00:15,055 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:00:15,056 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:00:15,062 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:00:15,323 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:00:15,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:16,324 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:00:16,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:17,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:18,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:19,096 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:19,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:20,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:21,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:00:21,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:00:21,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:22,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:23,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:24,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:24,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:25,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:26,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:27,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:28,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:29,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:30,619 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:30,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:31,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:32,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:33,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:34,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:35,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:35,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:36,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:00:36,537 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:00:36,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:37,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:38,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:39,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:40,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:41,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:41,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:42,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:43,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:44,296 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:00:44,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:45,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:46,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:47,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:47,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:48,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:00:48,633 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:00:48,633 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:00:48,636 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:00:49,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:49,354 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:00:50,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:50,355 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:00:51,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:51,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:00:51,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:00:52,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:52,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:53,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:54,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:55,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:56,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:57,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:58,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:00:58,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:00:59,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:00,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:01,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:02,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:03,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:04,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:04,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:05,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:06,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:06,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:01:06,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:01:07,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:08,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:09,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:10,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:10,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:11,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:12,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:13,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:14,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:14,291 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:01:15,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:15,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:16,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:17,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:18,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:18,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:01:18,711 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:01:18,712 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:01:18,716 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:01:19,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:19,381 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:01:20,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:20,382 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:01:21,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:21,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:01:21,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:01:21,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:22,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:23,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:24,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:25,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:26,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:27,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:27,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:28,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:29,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:30,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:31,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:32,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:32,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:33,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:34,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:35,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:36,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:36,537 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:01:36,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:01:37,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:37,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:38,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:39,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:40,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:41,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:42,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:43,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:43,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:44,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:44,295 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:01:45,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:46,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:47,123 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:48,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:48,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:49,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:50,133 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:51,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:51,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:01:51,539 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:01:51,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:01:51,665 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:01:51,665 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:01:51,670 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:01:52,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:52,407 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:01:52,408 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:01:53,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:53,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:01:54,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:55,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:56,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:57,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:58,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:59,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:01:59,628 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:00,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:01,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:02,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:03,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:04,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:04,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:05,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:06,172 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:06,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:02:06,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:02:07,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:08,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:09,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:10,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:10,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:11,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:12,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:13,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:14,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:14,299 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:02:15,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:16,180 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:16,630 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:17,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:18,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:19,183 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:20,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:21,180 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:21,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:02:21,538 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:02:21,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:02:21,812 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:02:21,813 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:02:21,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:21,818 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:02:22,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:22,439 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:02:22,439 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:02:23,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:24,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:25,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:26,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:27,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:27,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:28,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:29,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:30,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:31,233 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:32,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:33,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:33,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:34,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:35,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:36,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:36,538 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:02:36,539 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:02:37,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:38,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:39,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:39,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:40,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:41,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:42,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:43,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:44,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:44,303 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:02:44,630 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:45,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:46,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:47,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:48,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:49,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:49,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:50,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:51,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:51,551 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:02:51,554 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:02:52,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:53,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:54,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:54,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:02:54,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:02:54,984 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:02:54,984 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:02:54,989 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:02:55,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:55,467 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:02:56,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:56,468 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:02:57,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:58,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:02:59,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:00,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:00,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:01,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:02,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:03,402 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:04,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:05,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:06,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:06,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:03:06,545 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:03:06,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:07,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:08,403 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:09,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:10,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:11,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:11,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:12,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:13,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:14,307 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:03:14,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:15,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:16,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:17,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:17,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:18,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:19,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:20,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:21,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:03:21,545 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:03:21,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:22,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:22,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:23,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:24,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:25,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:26,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:27,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:28,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:03:28,548 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:03:28,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:03:28,552 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:28,553 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:03:28,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:29,498 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:03:29,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:30,499 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:03:30,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:31,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:32,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:33,642 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:33,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:34,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:35,695 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:36,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:03:36,545 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:03:36,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:37,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:38,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:39,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:39,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:40,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:41,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:42,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:43,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:44,311 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:03:44,642 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:44,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:45,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:46,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:47,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:48,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:49,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:49,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:50,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:51,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:03:51,545 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:03:51,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:52,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:53,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:54,649 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:03:54,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:55,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:56,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:57,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:57,934 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:03:57,935 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:03:57,936 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:03:57,937 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:03:58,524 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:03:58,525 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:03:58,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:03:59,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:00,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:00,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:01,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:02,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:03,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:04,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:05,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:06,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:04:06,545 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:04:06,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:06,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:07,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:08,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:09,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:10,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:11,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:11,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:12,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:13,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:14,315 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:04:14,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:15,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:16,644 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:16,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:17,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:18,799 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:19,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:20,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:21,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:04:21,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:04:21,800 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:22,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:22,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:23,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:24,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:25,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:26,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:27,652 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:27,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:04:27,671 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:04:27,671 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:04:27,673 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:04:28,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:28,551 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:04:28,552 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:04:28,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:29,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:30,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:31,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:32,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:33,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:33,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:34,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:35,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:36,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:04:36,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:04:36,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:37,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:38,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:39,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:39,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:40,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:41,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:42,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:43,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:44,319 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:04:44,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:44,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:45,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:46,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:47,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:48,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:49,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:49,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:50,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:51,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:04:51,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:04:51,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:52,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:53,834 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:54,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:04:54,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:55,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:56,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:57,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:58,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:04:58,619 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:04:58,619 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:04:58,621 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:04:59,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:04:59,577 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:04:59,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:00,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:00,578 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:05:01,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:02,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:03,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:04,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:05,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:05,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:06,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:06,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:05:06,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:05:07,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:08,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:09,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:10,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:11,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:11,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:12,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:13,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:14,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:14,323 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:05:15,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:16,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:16,655 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:17,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:18,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:19,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:20,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:21,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:21,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:05:21,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:05:22,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:22,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:23,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:24,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:25,038 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:26,044 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:27,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:27,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:28,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:29,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:29,331 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:05:29,332 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:05:29,333 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:05:29,334 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:05:29,607 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:05:30,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:30,608 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:05:31,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:32,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:32,745 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:33,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:34,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:35,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:36,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:36,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:05:36,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:05:37,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:38,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:38,657 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:39,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:40,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:41,074 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:42,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:43,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:43,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:44,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:44,327 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:05:45,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:46,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:47,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:48,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:48,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:49,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:50,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:51,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:51,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:05:51,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:05:52,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:53,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:54,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:54,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:05:55,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:56,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:57,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:58,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:58,577 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:05:58,578 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:05:58,578 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:05:58,579 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:05:58,632 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:05:59,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:05:59,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:00,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:00,634 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:06:01,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:02,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:03,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:04,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:05,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:05,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:06,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:06,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:06:06,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:06:07,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:08,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:09,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:10,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:10,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:11,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:12,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:13,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:14,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:14,330 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:06:15,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:16,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:16,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:17,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:18,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:19,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:20,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:21,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:21,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:06:21,546 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:06:22,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:22,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:23,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:24,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:25,090 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:26,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:27,091 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:27,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:28,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:06:28,075 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:06:28,075 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:06:28,077 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:06:28,485 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:28,660 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:06:28,660 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:06:29,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:30,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:31,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:32,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:33,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:33,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:34,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:35,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:36,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:36,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:06:36,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:06:37,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:38,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:39,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:39,665 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:40,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:41,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:42,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:43,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:44,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:44,334 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:06:44,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:45,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:46,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:47,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:48,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:49,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:50,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:50,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:51,221 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:51,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:06:51,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:06:52,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:53,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:54,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:55,228 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:55,757 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:06:56,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:57,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:57,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:06:57,796 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:06:57,797 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:06:57,798 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:06:58,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:06:58,695 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:06:58,696 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:06:59,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:00,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:01,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:01,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:02,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:03,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:04,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:05,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:06,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:06,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:07:06,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:07:07,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:07,644 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:08,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:09,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:10,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:11,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:12,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:12,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:13,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:14,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:14,338 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:07:15,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:16,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:17,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:18,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:18,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:19,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:20,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:21,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:21,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:07:21,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:07:22,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:23,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:24,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:24,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:25,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:26,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:27,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:28,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:28,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:07:28,348 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:07:28,349 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:07:28,350 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:07:28,728 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:07:29,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:29,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:30,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:30,729 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:07:31,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:32,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:33,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:34,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:34,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:35,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:36,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:36,546 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:07:36,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:07:37,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:38,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:39,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:40,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:40,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:41,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:42,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:43,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:44,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:44,343 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:07:45,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:45,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:46,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:47,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:48,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:49,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:50,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:51,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:51,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:07:51,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:07:51,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:52,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:53,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:54,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:55,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:56,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:56,774 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:07:57,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:58,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:58,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:07:58,888 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:07:58,888 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:07:58,890 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:07:59,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:07:59,757 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:08:00,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:00,758 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:08:01,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:02,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:02,767 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:03,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:04,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:05,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:06,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:06,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:08:06,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:08:07,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:08,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:08,659 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:09,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:10,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:11,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:12,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:13,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:13,767 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:14,296 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:14,346 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:08:15,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:16,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:17,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:18,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:18,767 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:19,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:20,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:21,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:21,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:08:21,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:08:22,297 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:23,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:24,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:24,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:25,303 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:26,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:27,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:28,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:28,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:08:28,639 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:08:28,639 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:08:28,640 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:08:28,781 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:08:29,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:29,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:30,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:30,783 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:08:31,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:32,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:33,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:34,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:34,771 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:35,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:36,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:36,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:08:36,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:08:37,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:38,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:39,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:40,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:40,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:41,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:42,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:43,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:44,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:44,350 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:08:45,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:45,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:46,305 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:47,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:48,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:49,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:50,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:51,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:51,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:08:51,547 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:08:51,676 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:52,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:53,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:54,314 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:55,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:56,314 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:56,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:08:57,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:58,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:58,868 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:08:58,869 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:08:58,869 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:08:58,871 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:08:59,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:08:59,803 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:09:00,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:00,804 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:09:01,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:02,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:02,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:03,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:04,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:05,315 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:06,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:06,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:09:06,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:09:07,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:08,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:08,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:09,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:10,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:11,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:12,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:13,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:13,777 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:14,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:14,354 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:09:15,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:16,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:17,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:18,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:19,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:19,776 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:20,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:21,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:21,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:09:21,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:09:22,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:23,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:24,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:24,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:25,321 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:26,321 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:27,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:28,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:29,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:30,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:09:30,304 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:09:30,304 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:09:30,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:30,306 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:09:30,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:30,836 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:09:30,837 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:09:31,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:32,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:33,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:34,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:35,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:35,781 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:36,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:36,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:09:36,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:09:37,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:38,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:39,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:40,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:41,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:41,781 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:42,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:43,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:44,357 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:09:44,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:45,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:46,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:46,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:47,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:48,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:49,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:50,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:51,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:51,547 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:09:51,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:09:52,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:52,656 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:53,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:54,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:55,498 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:56,499 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:57,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:57,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:09:58,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:09:59,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:00,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:01,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:02,432 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:10:02,434 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:10:02,434 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:10:02,435 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:10:02,866 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:10:02,866 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:10:03,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:03,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:04,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:05,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:06,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:06,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:10:06,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:10:07,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:08,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:08,664 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:09,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:10,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:11,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:12,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:13,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:13,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:14,361 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:10:14,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:15,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:16,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:17,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:18,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:18,785 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:19,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:20,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:21,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:21,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:10:21,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:10:22,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:23,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:24,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:24,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:25,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:26,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:27,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:28,375 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:29,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:29,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:30,380 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:31,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:32,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:32,663 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:10:32,665 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:10:32,666 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:10:32,667 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:10:32,898 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:10:33,694 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:34,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:34,899 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:10:35,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:35,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:36,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:10:36,548 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:10:36,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:37,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:38,696 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:39,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:40,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:41,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:41,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:42,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:43,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:44,365 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:10:44,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:45,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:46,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:46,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:47,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:48,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:49,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:50,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:51,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:10:51,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:10:51,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:52,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:52,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:53,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:54,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:55,698 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:56,702 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:57,699 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:57,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:10:58,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:10:59,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:00,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:01,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:01,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:11:01,711 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:11:01,712 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:11:01,713 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:11:01,930 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:11:02,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:02,931 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:11:03,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:03,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:04,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:05,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:06,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:11:06,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:11:06,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:07,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:08,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:09,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:09,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:10,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:11,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:12,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:13,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:14,369 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:11:14,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:14,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:15,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:17,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:18,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:19,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:19,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:20,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:21,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:21,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:11:21,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:11:22,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:23,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:24,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:25,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:25,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:26,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:27,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:28,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:29,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:30,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:30,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:31,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:32,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:33,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:34,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:34,673 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:11:34,674 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:11:34,674 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:11:34,675 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:11:34,964 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:11:35,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:36,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:36,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:36,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:11:36,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:11:36,966 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:11:37,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:38,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:39,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:40,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:41,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:42,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:42,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:43,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:44,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:44,373 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:11:45,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:46,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:47,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:47,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:48,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:49,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:50,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:51,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:51,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:11:51,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:11:52,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:52,665 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:53,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:54,087 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:55,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:56,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:57,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:58,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:11:58,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:11:59,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:00,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:01,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:02,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:02,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:12:02,913 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:12:02,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:12:02,914 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:12:02,991 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:12:03,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:03,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:04,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:04,992 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:12:05,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:06,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:12:06,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:12:06,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:07,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:08,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:09,662 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:09,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:10,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:11,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:12,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:13,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:14,377 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:12:14,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:15,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:15,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:16,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:17,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:18,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:19,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:20,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:20,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:21,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:12:21,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:12:21,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:22,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:23,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:24,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:25,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:26,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:26,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:27,959 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:28,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:29,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:30,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:31,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:31,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:32,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:33,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:34,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:12:34,520 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:12:34,521 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:12:34,523 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:12:34,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:35,023 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:12:35,024 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:12:35,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:36,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:12:36,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:12:36,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:36,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:37,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:38,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:39,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:40,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:41,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:42,091 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:42,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:43,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:44,381 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:12:44,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:45,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:46,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:48,092 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:48,153 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:49,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:50,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:51,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:51,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:12:51,549 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:12:52,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:53,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:53,652 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:12:54,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:55,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:56,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:57,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:58,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:59,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:12:59,092 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:00,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:01,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:02,056 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:03,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:04,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:04,095 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:05,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:06,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:06,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:13:06,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:13:06,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:13:06,747 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:13:06,748 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:13:06,749 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:13:07,059 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:13:07,060 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:13:07,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:08,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:09,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:09,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:10,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:11,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:12,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:13,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:14,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:14,386 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:13:15,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:15,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:16,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:17,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:18,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:19,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:20,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:20,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:21,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:21,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:13:21,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:13:22,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:23,158 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:24,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:25,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:26,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:26,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:27,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:28,159 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:29,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:30,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:31,097 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:31,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:32,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:33,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:34,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:34,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:13:34,904 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:13:34,905 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:13:34,907 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:13:35,088 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:13:35,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:36,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:36,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:36,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:13:36,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:13:37,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:38,090 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:13:38,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:39,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:40,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:41,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:42,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:42,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:43,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:44,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:44,389 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:13:45,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:46,316 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:47,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:48,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:48,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:49,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:50,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:51,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:51,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:13:51,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:13:52,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:53,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:53,653 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:54,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:55,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:56,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:57,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:58,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:13:59,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:13:59,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:00,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:01,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:02,322 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:03,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:04,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:04,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:05,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:05,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:14:05,396 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:14:05,396 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:14:05,398 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:14:06,124 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:14:06,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:06,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:14:06,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:14:07,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:08,126 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:14:08,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:09,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:09,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:10,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:11,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:12,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:13,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:14,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:14,393 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:14:15,322 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:15,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:16,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:17,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:18,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:19,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:20,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:20,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:21,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:21,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:14:21,550 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:14:22,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:23,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:24,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:25,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:26,322 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:26,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:27,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:28,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:29,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:30,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:31,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:32,323 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:32,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:33,329 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:34,329 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:35,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:36,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:36,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:14:36,556 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:14:37,334 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:37,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:38,117 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:14:38,118 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:14:38,119 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:14:38,120 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:14:38,148 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:14:38,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:39,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:40,149 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:14:40,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:41,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:42,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:43,327 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:43,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:44,396 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:14:44,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:45,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:46,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:47,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:48,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:49,327 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:49,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:50,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:51,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:51,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:14:51,555 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:14:52,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:53,535 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:54,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:54,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:14:55,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:56,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:57,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:58,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:14:59,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:00,327 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:00,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:01,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:02,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:03,539 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:04,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:05,332 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:05,542 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:06,542 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:06,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:15:06,560 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:15:07,544 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:08,543 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:09,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:15:09,348 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:15:09,348 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:15:09,349 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:15:09,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:10,183 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:15:10,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:10,760 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:11,760 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:12,186 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:15:12,760 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:13,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:14,400 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:15:14,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:15,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:16,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:16,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:17,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:18,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:19,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:20,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:21,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:15:21,558 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:15:21,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:21,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:22,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:23,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:24,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:25,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:26,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:27,333 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:27,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:28,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:29,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:30,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:31,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:32,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:33,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:33,770 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:34,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:35,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:36,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:15:36,562 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:15:36,768 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:37,768 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:38,661 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:38,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:39,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:15:39,028 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:15:39,028 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:15:39,030 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:15:39,209 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:15:39,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:40,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:41,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:42,211 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:15:42,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:43,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:44,339 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:44,403 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:15:44,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:45,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:46,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:47,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:48,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:49,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:49,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:50,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:51,561 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:15:51,562 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:15:51,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:52,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:53,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:54,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:15:54,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:55,805 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:56,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:57,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:58,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:15:59,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:00,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:00,817 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:01,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:02,821 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:05,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:05,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:06,224 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:16:06,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:10,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:13,628 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:16:13,629 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:16:14,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:14,407 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:16:15,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:20,545 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:16:20,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:20,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:21,876 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:22,658 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:16:22,659 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:16:25,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:28,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:29,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:16:30,081 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:30,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:30,867 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:35,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:37,597 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:16:38,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:39,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:16:39,384 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:16:40,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:41,519 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:44,411 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:16:46,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:16:47,303 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:47,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:48,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:52,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:16:56,307 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:16:56,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:16:57,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:00,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:17:00,824 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:17:00,864 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:00,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:01,880 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:17:01,955 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:02,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:02,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:03,881 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:17:03,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:04,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:05,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:06,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:07,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:07,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:08,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:09,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:10,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:11,307 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:17:11,308 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:17:11,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:12,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:13,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:13,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:14,415 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:17:14,956 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:15,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:16,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:17,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:18,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:19,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:19,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:20,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:21,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:22,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:23,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:24,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:24,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:25,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:26,307 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:17:26,308 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:17:26,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:27,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:28,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:29,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:29,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:30,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:31,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:32,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:33,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:34,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:34,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:35,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:36,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:37,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:38,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:39,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:39,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:40,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:41,307 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:17:41,308 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:17:41,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:42,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:43,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:44,419 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:17:44,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:45,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:45,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:46,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:47,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:48,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:49,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:50,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:17:50,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:51,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:52,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:53,965 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:17:55,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:00,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:05,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:10,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:13,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:18:13,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:18:13,952 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:13,953 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:18:14,423 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:18:14,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:15,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:15,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:16,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:17,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:18,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:19,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:20,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:20,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:21,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:22,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:23,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:24,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:25,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:25,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:26,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:27,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:28,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:28,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:18:28,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:18:29,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:30,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:31,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:31,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:32,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:33,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:34,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:35,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:36,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:36,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:37,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:38,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:39,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:40,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:41,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:41,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:42,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:43,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:43,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:18:43,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:18:44,427 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:18:44,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:45,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:46,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:46,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:47,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:48,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:49,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:50,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:51,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:51,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:52,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:53,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:54,915 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:55,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:56,431 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:18:56,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:57,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:58,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:18:58,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:18:58,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:18:59,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:00,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:01,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:02,005 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:02,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:03,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:04,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:05,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:06,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:07,005 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:07,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:08,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:09,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:10,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:11,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:12,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:12,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:13,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:19:13,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:19:13,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:14,431 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:19:14,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:15,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:16,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:17,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:17,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:18,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:19,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:20,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:21,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:22,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:22,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:23,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:24,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:25,918 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:26,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:27,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:27,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:28,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:19:28,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:19:28,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:29,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:30,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:31,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:32,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:33,081 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:33,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:34,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:35,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:36,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:37,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:38,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:38,919 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:39,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:40,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:41,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:42,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:43,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:43,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:19:43,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:19:43,957 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:44,435 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:19:44,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:45,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:46,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:47,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:48,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:48,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:49,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:50,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:51,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:52,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:53,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:53,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:54,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:55,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:56,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:57,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:58,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:19:58,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:19:58,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:19:58,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:19:59,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:00,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:01,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:02,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:03,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:04,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:04,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:05,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:06,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:07,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:08,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:09,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:09,922 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:14,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:14,439 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:20:19,441 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:24,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:29,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:34,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:39,444 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:44,442 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:20:44,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 10:20:44,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:44,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:20:44,711 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:20:44,833 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:45,676 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:46,676 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:47,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:48,297 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:20:48,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:49,298 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:20:49,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:50,326 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:50,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:51,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:52,300 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:20:52,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:53,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:54,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:55,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:20:55,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:56,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:57,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:58,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:20:59,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:20:59,670 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:20:59,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:00,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:00,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:01,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:02,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:03,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:04,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:05,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:05,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:06,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:07,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:08,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:09,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:10,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:10,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:11,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:12,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:13,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:14,444 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:21:14,674 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:21:14,681 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:21:14,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:15,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:15,784 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:16,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:17,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:18,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:21:18,153 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:21:18,153 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:21:18,154 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:21:18,330 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:21:18,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:19,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:20,332 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:21:20,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:21,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:21,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:22,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:23,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:24,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:25,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:26,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:26,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:27,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:28,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:29,675 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:21:29,675 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:21:29,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:30,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:31,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:31,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:32,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:33,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:34,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:35,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:36,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:36,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:37,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:38,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:39,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:40,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:41,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:41,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:42,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:43,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:44,456 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:21:44,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:21:44,682 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:21:44,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:45,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:46,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:46,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:47,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:48,133 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:21:48,134 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:21:48,135 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:21:48,136 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:21:48,361 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:21:48,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:49,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:50,363 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:21:50,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:51,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:52,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:52,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:53,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:54,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:55,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:56,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:57,175 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:21:57,724 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:58,725 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:21:59,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:21:59,679 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:21:59,937 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:00,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:01,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:02,772 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:02,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:03,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:04,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:05,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:06,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:07,773 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:07,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:08,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:09,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:10,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:11,863 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:12,778 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:12,857 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:13,857 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:14,463 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:22:14,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:22:14,682 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:22:14,857 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:15,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:16,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:17,497 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:22:17,498 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:22:17,499 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:22:17,500 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:22:17,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:18,390 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:22:18,390 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:22:18,539 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:18,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:19,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:20,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:21,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:22,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:23,539 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:23,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:24,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:25,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:26,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:27,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:28,540 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:28,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:29,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:22:29,682 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:22:29,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:30,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:31,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:32,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:33,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:33,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:34,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:35,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:36,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:37,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:38,788 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:38,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:39,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:40,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:41,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:42,903 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:43,795 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:43,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:44,461 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:22:44,688 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:22:44,688 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:22:44,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:45,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:46,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:22:46,810 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:22:46,811 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:22:46,815 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:22:47,227 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:47,418 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:22:47,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:48,419 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:22:48,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:48,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:49,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:50,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:51,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:52,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:53,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:53,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:54,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:55,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:56,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:57,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:58,853 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:22:58,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:22:59,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:22:59,683 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:22:59,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:00,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:01,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:02,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:03,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:04,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:04,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:05,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:06,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:07,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:08,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:09,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:09,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:10,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:11,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:12,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:13,963 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:14,462 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:23:14,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:23:14,683 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:23:14,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:14,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:15,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:16,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:17,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:18,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:19,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:19,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:20,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:23:20,427 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:23:20,428 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:23:20,431 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:23:20,446 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:23:20,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:21,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:22,448 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:23:22,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:23,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:24,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:25,468 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:25,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:26,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:27,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:28,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:29,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:23:29,683 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:23:29,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:30,764 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:30,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:31,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:33,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:34,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:35,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:35,764 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:36,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:37,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:38,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:39,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:40,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:40,765 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:41,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:42,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:43,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:44,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:44,466 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:23:44,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:23:44,683 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:23:45,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:45,782 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:46,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:47,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:48,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:49,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:50,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:50,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:51,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:51,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:23:51,667 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:23:51,668 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:23:51,670 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:23:52,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:52,478 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:23:53,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:54,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:54,480 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:23:55,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:56,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:56,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:23:57,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:58,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:59,239 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:23:59,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:23:59,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:24:00,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:01,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:01,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:02,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:03,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:04,448 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:05,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:06,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:06,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:07,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:08,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:09,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:10,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:11,350 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:11,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:12,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:13,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:14,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:14,470 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:24:14,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:24:14,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:24:15,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:16,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:17,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:17,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:18,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:19,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:20,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:21,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:22,000 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:24:22,001 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:24:22,001 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:24:22,003 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:24:22,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:22,504 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:24:23,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:23,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:24,410 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:24,505 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:24:25,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:26,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:27,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:28,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:28,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:29,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:29,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:24:29,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:24:30,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:31,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:32,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:33,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:33,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:34,411 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:35,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:36,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:37,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:38,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:38,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:39,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:40,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:41,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:42,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:43,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:43,814 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:44,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:44,474 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:24:44,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:24:44,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:24:45,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:46,412 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:47,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:48,416 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:49,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:49,416 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:50,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:51,421 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:52,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:53,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:24:53,053 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:24:53,054 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:24:53,057 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:24:53,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:53,531 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:24:54,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:54,532 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:24:55,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:24:55,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:56,463 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:57,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:58,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:59,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:24:59,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:24:59,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:25:00,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:00,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:01,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:02,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:03,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:04,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:05,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:05,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:06,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:07,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:08,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:09,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:10,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:10,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:11,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:12,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:13,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:14,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:14,478 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:25:14,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:25:14,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:25:15,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:15,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:16,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:17,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:18,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:19,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:20,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:20,805 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:21,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:22,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:23,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:24,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:25,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:25,808 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:25,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:25:25,907 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:25:25,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:25:25,909 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:25:26,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:26,560 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:25:27,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:28,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:28,561 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:25:29,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:29,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:25:29,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:25:30,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:30,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:31,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:32,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:33,471 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:34,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:35,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:35,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:36,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:37,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:38,472 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:39,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:40,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:40,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:41,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:42,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:43,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:44,482 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:25:44,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:44,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:25:44,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:25:45,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:46,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:46,757 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:47,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:48,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:49,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:50,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:51,612 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:51,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:52,605 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:53,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:54,606 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:55,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:25:55,087 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:25:55,088 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:25:55,089 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:25:55,584 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:25:55,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:56,585 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:25:56,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:57,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:25:57,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:58,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:59,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:25:59,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:25:59,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:26:00,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:01,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:02,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:02,789 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:03,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:04,607 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:05,608 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:06,846 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:07,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:07,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:08,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:09,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:10,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:11,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:12,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:12,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:13,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:14,486 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:26:14,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:14,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:26:14,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:26:15,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:16,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:17,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:18,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:18,793 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:19,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:20,617 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:21,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:22,621 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:23,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:23,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:24,435 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:26:24,436 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:26:24,436 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:26:24,437 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:26:24,615 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:26:24,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:25,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:26,616 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:26:26,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:27,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:28,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:29,476 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:29,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:26:29,684 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:26:29,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:30,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:31,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:32,949 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:33,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:34,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:34,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:35,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:36,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:37,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:38,961 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:39,775 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:39,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:40,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:41,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:42,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:43,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:44,491 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:26:44,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:26:44,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:26:44,845 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:44,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:45,962 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:46,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:47,964 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:48,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:49,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:49,966 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:50,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:51,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:52,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:26:52,455 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:26:52,455 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:26:52,457 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:26:52,639 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:26:52,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:53,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:54,641 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:26:54,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:55,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:26:55,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:56,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:57,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:58,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:26:59,684 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:26:59,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:26:59,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:00,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:00,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:01,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:02,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:03,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:04,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:05,838 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:05,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:06,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:07,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:08,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:09,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:10,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:10,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:11,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:12,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:13,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:14,494 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:27:14,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:27:14,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:27:14,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:15,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:16,786 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:16,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:17,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:18,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:19,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:20,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:21,787 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:21,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:22,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:23,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:24,977 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:25,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:26,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:26,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:27,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:28,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:27:28,112 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:27:28,113 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:27:28,114 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:27:28,672 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:27:28,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:29,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:27:29,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:27:29,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:30,674 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:27:30,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:31,848 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:31,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:32,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:33,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:34,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:35,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:36,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:36,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:37,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:38,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:39,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:40,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:41,849 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:41,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:42,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:43,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:44,498 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:27:44,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:27:44,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:27:44,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:45,974 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:46,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:47,847 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:47,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:48,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:49,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:50,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:51,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:52,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:52,975 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:53,978 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:54,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:55,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:56,978 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:27:56,979 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:27:56,980 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:27:56,981 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:27:57,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:57,698 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:27:57,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:58,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:27:58,699 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:27:58,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:27:59,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:27:59,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:27:59,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:00,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:01,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:02,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:03,753 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:03,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:04,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:05,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:06,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:07,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:08,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:08,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:09,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:10,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:11,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:12,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:13,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:13,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:14,502 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:28:14,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:28:14,685 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:28:14,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:15,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:16,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:17,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:18,826 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:18,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:19,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:20,984 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:21,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:22,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:23,837 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:23,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:24,986 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:25,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:26,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:27,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:28,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:28,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:29,473 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:28:29,474 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:28:29,475 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:28:29,476 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:28:29,728 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:28:29,888 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:28:29,889 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:28:29,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:30,729 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:28:30,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:31,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:32,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:33,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:34,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:34,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:35,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:36,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:37,992 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:38,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:39,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:39,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:40,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:41,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:42,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:43,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:44,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:44,506 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:28:44,883 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:28:44,884 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:28:44,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:45,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:46,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:47,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:48,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:49,946 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:49,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:50,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:51,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:52,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:53,997 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:54,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:28:54,997 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:55,998 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:56,998 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:58,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:59,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:28:59,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:28:59,064 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:28:59,065 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:28:59,066 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:28:59,755 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:28:59,883 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:28:59,884 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:29:00,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:00,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:00,756 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:29:01,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:02,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:03,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:04,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:05,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:05,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:06,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:07,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:08,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:09,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:10,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:10,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:11,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:12,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:13,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:14,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:14,509 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:29:14,883 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:29:14,884 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:29:15,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:15,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:16,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:17,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:18,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:19,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:20,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:20,950 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:21,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:22,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:23,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:24,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:25,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:25,951 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:26,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:27,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:28,008 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:29,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:29,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:29:29,887 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:29:30,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:30,977 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:31,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:32,012 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:32,771 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:29:32,772 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:29:32,772 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:29:32,774 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:29:32,784 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:29:33,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:34,786 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:29:34,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:35,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:36,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:36,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:37,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:38,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:39,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:40,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:41,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:41,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:42,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:43,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:44,514 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:29:44,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:44,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:29:44,887 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:29:45,810 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:46,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:47,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:47,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:48,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:49,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:50,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:51,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:52,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:52,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:53,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:54,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:55,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:56,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:57,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:29:57,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:58,820 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:59,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:29:59,890 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:29:59,890 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:30:00,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:01,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:02,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:02,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:03,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:04,562 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:30:04,563 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:30:04,563 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:30:04,565 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:30:04,817 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:30:04,978 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:05,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:06,819 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:30:06,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:07,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:08,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:08,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:09,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:10,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:11,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:12,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:13,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:13,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:14,517 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:30:14,890 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:30:14,891 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:30:14,976 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:16,147 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:17,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:18,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:19,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:19,059 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:20,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:21,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:22,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:23,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:24,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:24,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:25,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:26,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:27,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:28,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:29,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:29,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:29,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:30:29,897 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:30:30,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:31,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:32,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:32,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:30:32,385 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:30:32,385 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:30:32,387 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:30:32,847 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:30:33,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:34,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:34,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:34,849 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:30:35,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:36,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:37,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:38,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:39,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:39,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:40,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:41,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:42,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:43,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:44,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:44,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:44,521 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:30:44,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:30:44,897 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:30:45,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:46,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:47,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:48,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:49,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:49,967 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:50,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:51,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:52,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:53,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:54,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:54,968 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:30:55,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:56,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:57,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:58,480 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:59,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:30:59,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:30:59,917 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:31:00,075 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:00,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:01,487 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:02,484 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:03,484 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:04,485 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:05,080 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:05,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:31:05,210 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:31:05,210 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:31:05,212 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:31:05,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:05,883 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:31:06,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:06,884 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:31:07,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:08,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:09,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:10,250 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:10,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:11,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:12,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:13,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:14,524 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:31:14,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:14,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:31:14,902 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:31:15,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:16,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:16,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:17,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:18,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:19,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:20,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:21,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:21,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:22,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:23,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:24,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:25,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:26,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:26,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:27,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:28,628 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:29,628 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:29,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:31:29,901 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:31:30,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:31,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:32,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:32,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:33,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:34,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:35,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:35,856 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:31:35,858 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:31:35,859 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:31:35,860 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:31:35,911 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:31:36,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:36,912 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:31:37,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:37,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:38,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:39,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:40,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:41,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:42,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:42,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:43,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:44,528 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:31:44,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:44,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:31:44,901 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:31:45,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:46,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:47,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:47,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:48,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:49,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:50,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:51,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:52,633 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:52,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:53,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:54,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:55,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:56,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:57,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:57,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:31:58,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:59,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:31:59,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:31:59,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:32:00,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:01,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:02,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:02,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:03,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:04,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:05,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:32:05,065 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:32:05,065 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:32:05,066 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:32:05,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:05,937 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:32:06,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:06,938 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:32:07,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:08,104 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:08,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:09,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:10,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:11,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:12,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:13,105 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:13,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:14,532 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:32:14,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:14,906 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:32:14,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:32:15,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:16,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:17,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:18,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:19,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:19,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:20,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:21,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:22,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:23,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:24,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:24,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:25,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:26,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:27,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:28,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:29,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:29,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:29,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:32:29,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:32:30,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:31,643 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:32,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:33,648 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:34,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:34,649 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:35,646 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:36,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:32:36,612 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:32:36,612 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:32:36,613 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:32:36,966 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:32:37,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:38,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:38,967 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:32:39,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:39,651 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:40,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:41,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:42,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:43,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:44,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:44,536 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:32:44,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:32:44,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:32:44,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:45,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:46,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:47,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:48,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:49,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:49,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:50,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:51,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:52,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:53,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:54,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:54,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:32:55,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:56,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:57,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:58,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:59,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:32:59,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:32:59,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:33:00,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:00,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:01,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:02,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:03,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:04,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:05,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:05,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:06,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:07,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:07,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:33:07,764 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:33:07,764 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:33:07,765 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:33:07,999 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:33:08,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:09,000 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:33:09,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:10,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:10,805 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:11,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:12,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:13,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:14,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:14,540 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:33:14,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:33:14,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:33:15,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:16,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:16,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:17,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:18,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:19,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:20,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:21,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:21,213 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:22,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:23,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:24,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:25,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:26,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:26,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:27,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:28,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:29,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:29,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:33:29,907 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:33:30,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:31,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:32,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:32,216 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:33,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:34,214 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:35,218 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:36,215 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:37,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:37,215 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:37,940 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:33:37,942 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:33:37,942 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:33:37,943 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:33:38,030 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:33:38,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:39,031 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:33:39,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:40,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:41,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:42,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:42,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:43,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:44,359 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:44,544 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:33:44,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:33:44,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:33:45,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:46,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:47,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:48,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:48,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:49,360 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:50,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:51,449 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:52,449 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:53,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:53,449 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:54,449 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:55,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:56,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:57,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:58,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:33:58,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:59,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:33:59,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:33:59,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:34:00,450 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:01,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:02,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:03,097 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:03,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:04,456 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:05,456 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:06,456 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:07,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:34:07,065 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:34:07,065 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:34:07,067 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:34:07,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:08,067 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:34:08,105 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:08,476 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:09,068 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:34:09,476 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:10,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:11,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:12,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:13,106 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:13,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:14,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:14,548 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:34:14,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:34:14,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:34:15,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:16,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:17,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:18,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:19,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:19,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:20,477 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:21,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:22,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:23,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:24,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:24,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:25,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:26,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:27,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:28,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:29,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:29,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:29,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:34:29,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:34:30,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:31,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:32,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:33,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:34,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:34,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:35,482 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:36,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:37,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:38,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:39,080 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:39,284 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:34:39,285 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:34:39,285 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:34:39,287 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:34:39,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:40,093 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:34:40,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:41,094 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:34:41,703 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:42,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:43,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:44,325 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:44,552 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:34:44,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:44,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:34:44,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:34:45,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:46,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:47,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:48,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:49,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:49,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:50,704 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:51,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:52,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:53,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:54,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:54,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:34:55,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:56,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:57,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:58,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:59,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:34:59,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:34:59,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:35:00,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:00,705 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:01,706 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:02,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:03,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:04,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:05,058 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:05,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:06,714 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:07,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:08,076 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:35:08,077 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:35:08,078 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:35:08,079 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:35:08,120 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:35:08,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:09,122 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:35:09,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:10,118 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:10,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:11,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:12,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:13,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:14,556 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:35:14,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:14,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:35:14,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:35:15,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:16,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:16,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:17,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:18,711 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:19,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:20,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:21,056 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:21,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:22,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:23,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:24,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:25,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:26,056 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:26,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:27,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:28,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:29,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:29,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:35:29,908 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:35:30,713 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:31,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:32,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:32,715 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:33,721 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:34,719 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:35,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:36,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:37,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:37,175 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:35:37,176 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:35:37,176 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:35:37,178 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:35:37,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:38,149 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:35:38,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:39,150 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:35:39,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:40,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:41,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:42,216 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:42,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:43,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:44,560 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:35:44,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:44,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:35:44,909 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:35:45,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:46,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:47,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:48,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:48,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:49,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:50,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:51,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:52,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:53,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:53,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:54,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:55,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:56,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:57,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:58,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:35:58,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:59,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:35:59,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:35:59,909 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:36:00,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:01,792 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:02,801 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:03,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:03,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:04,796 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:05,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:06,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:07,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:08,081 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:08,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:36:08,254 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:36:08,254 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:36:08,256 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:36:08,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:09,178 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:36:09,179 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:36:09,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:10,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:11,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:12,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:13,295 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:13,797 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:14,564 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:36:14,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:14,908 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:36:14,909 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:36:15,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:16,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:17,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:18,798 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:19,012 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:19,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:20,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:21,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:22,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:23,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:24,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:24,824 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:25,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:26,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:27,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:28,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:29,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:29,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:29,916 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:36:29,923 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:36:30,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:31,828 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:32,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:33,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:34,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:35,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:35,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:35,858 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:36:35,859 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:36:35,860 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:36:35,861 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:36:36,207 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:36:36,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:37,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:38,208 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:36:38,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:39,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:40,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:40,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:41,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:42,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:43,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:44,568 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:36:44,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:44,910 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:36:44,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:36:45,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:46,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:46,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:47,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:48,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:49,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:50,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:51,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:51,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:52,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:53,850 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:54,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:55,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:56,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:36:56,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:57,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:58,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:59,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:36:59,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:36:59,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:37:00,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:01,074 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:01,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:02,854 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:03,855 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:04,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:05,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:06,086 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:06,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:07,859 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:08,340 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:37:08,342 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:37:08,342 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:37:08,344 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:37:08,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:09,235 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:37:09,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:10,236 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:37:10,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:11,383 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:11,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:12,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:13,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:14,572 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:37:14,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:14,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:37:14,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:37:15,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:16,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:17,012 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:17,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:18,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:19,860 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:20,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:21,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:22,012 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:22,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:23,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:24,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:25,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:26,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:27,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:27,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:28,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:29,861 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:29,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:37:29,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:37:30,862 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:31,871 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:32,872 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:32,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:33,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:34,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:35,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:36,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:36,985 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:37:36,986 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:37:36,986 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:37:36,987 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:37:37,262 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:37:37,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:38,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:38,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:39,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:40,264 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:37:40,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:41,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:42,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:43,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:43,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:44,576 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:37:44,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:44,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:37:44,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:37:45,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:46,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:47,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:48,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:49,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:49,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:50,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:51,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:52,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:53,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:54,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:54,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:55,880 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:56,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:57,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:58,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:59,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:37:59,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:37:59,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:37:59,911 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:38:00,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:01,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:02,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:03,888 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:04,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:04,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:05,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:06,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:07,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:08,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:38:08,627 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:38:08,628 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:38:08,629 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:38:09,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:09,291 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:38:09,668 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:10,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:10,292 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:38:11,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:12,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:13,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:14,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:14,580 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:38:14,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:38:14,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:38:15,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:15,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:16,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:17,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:18,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:19,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:20,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:20,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:21,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:22,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:23,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:24,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:25,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:25,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:26,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:27,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:28,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:29,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:29,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:38:29,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:38:30,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:30,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:31,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:32,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:33,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:34,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:35,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:35,074 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:36,036 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:37,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:38,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:39,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:40,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:40,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:41,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:41,697 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:38:41,698 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:38:41,698 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:38:41,699 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:38:42,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:42,322 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:38:43,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:44,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:44,324 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:38:44,584 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:38:44,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:38:44,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:38:45,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:46,057 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:46,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:47,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:48,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:49,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:50,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:51,057 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:51,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:52,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:53,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:54,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:55,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:56,058 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:38:56,113 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:57,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:58,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:59,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:38:59,911 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:38:59,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:39:00,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:01,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:01,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:02,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:03,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:04,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:05,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:06,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:06,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:07,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:08,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:09,121 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:10,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:11,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:11,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:39:11,242 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:39:11,242 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:39:11,243 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:39:11,354 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:39:12,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:12,281 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:13,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:14,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:14,356 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:39:14,588 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:39:14,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:39:14,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:39:15,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:16,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:17,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:18,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:18,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:19,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:20,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:21,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:22,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:23,029 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:23,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:24,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:25,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:26,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:27,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:28,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:28,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:29,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:29,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:39:29,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:39:30,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:31,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:32,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:33,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:33,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:34,127 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:35,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:36,131 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:37,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:38,056 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:38,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:39,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:40,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:40,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:39:40,751 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:39:40,752 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:39:40,753 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:39:41,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:41,381 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:39:42,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:42,382 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:39:43,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:43,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:44,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:44,592 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:39:44,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:39:44,912 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:39:45,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:46,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:47,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:48,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:49,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:49,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:50,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:51,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:52,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:53,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:54,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:54,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:55,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:56,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:57,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:58,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:59,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:39:59,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:39:59,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:39:59,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:40:00,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:01,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:02,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:03,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:04,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:05,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:05,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:06,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:07,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:08,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:09,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:09,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:40:09,443 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:40:09,443 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:40:09,444 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:40:10,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:10,403 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:40:10,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:11,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:12,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:12,405 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:40:13,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:14,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:14,596 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:40:14,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:40:14,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:40:15,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:16,036 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:16,168 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:17,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:18,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:19,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:20,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:21,036 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:21,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:22,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:23,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:24,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:25,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:26,037 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:26,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:27,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:28,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:29,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:29,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:40:29,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:40:30,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:31,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:31,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:32,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:33,179 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:34,173 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:35,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:36,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:36,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:37,179 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:38,178 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:38,660 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:40:38,661 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:40:38,662 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:40:38,663 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:40:39,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:39,427 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:40:40,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:40,429 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:40:41,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:41,702 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:42,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:43,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:44,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:44,598 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:40:44,912 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:40:44,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:40:45,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:46,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:47,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:47,181 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:48,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:49,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:50,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:51,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:52,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:52,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:53,182 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:54,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:55,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:56,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:57,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:40:57,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:58,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:59,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:40:59,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:40:59,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:41:00,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:01,224 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:02,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:03,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:03,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:04,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:05,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:06,225 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:07,231 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:08,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:08,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:09,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:10,234 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:11,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:12,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:12,266 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:41:12,266 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:41:12,267 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:41:12,268 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:41:12,456 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:41:13,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:13,308 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:14,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:14,457 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:41:14,602 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:41:14,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:41:14,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:41:15,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:16,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:17,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:18,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:19,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:19,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:20,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:21,235 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:22,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:23,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:24,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:24,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:25,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:26,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:27,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:28,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:29,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:29,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:29,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:41:29,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:41:30,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:31,236 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:32,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:33,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:34,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:34,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:35,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:36,237 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:37,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:38,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:39,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:39,242 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:40,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:41,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:42,245 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:42,671 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:41:42,673 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:41:42,673 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:41:42,674 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:41:43,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:43,485 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:41:44,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:44,486 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:41:44,607 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:41:44,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:44,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:41:44,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:41:45,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:46,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:47,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:48,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:49,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:50,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:50,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:51,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:52,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:53,246 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:54,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:55,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:41:55,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:56,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:57,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:58,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:59,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:41:59,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:41:59,913 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:42:00,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:01,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:01,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:02,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:03,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:04,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:05,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:06,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:06,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:07,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:08,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:09,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:10,259 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:11,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:11,252 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:12,254 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:13,253 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:14,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:42:14,104 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:42:14,104 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:42:14,105 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:42:14,510 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:42:14,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:14,609 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:42:14,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:42:14,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:42:15,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:16,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:16,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:16,511 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:42:17,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:18,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:19,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:20,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:21,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:21,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:22,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:23,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:24,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:25,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:26,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:26,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:27,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:28,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:29,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:29,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:42:29,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:42:30,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:31,075 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:31,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:32,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:33,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:34,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:35,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:36,075 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:36,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:37,511 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:38,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:39,520 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:40,515 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:41,081 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:41,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:42,517 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:43,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:42:43,448 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:42:43,448 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:42:43,449 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:42:43,535 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:42:43,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:44,536 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:42:44,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:44,613 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:42:44,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:42:44,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:42:45,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:46,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:46,986 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:47,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:48,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:49,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:50,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:51,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:51,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:52,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:53,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:54,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:55,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:56,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:56,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:42:57,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:58,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:59,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:42:59,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:42:59,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:43:00,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:01,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:02,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:02,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:03,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:04,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:05,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:06,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:07,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:07,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:08,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:09,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:10,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:11,620 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:12,065 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:12,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:13,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:14,381 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:43:14,383 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:43:14,383 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:43:14,384 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:43:14,561 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:43:14,618 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:43:14,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:14,913 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:43:14,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:43:15,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:16,563 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:43:16,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:17,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:17,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:18,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:19,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:20,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:21,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:22,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:22,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:23,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:24,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:25,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:26,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:27,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:27,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:28,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:29,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:29,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:43:29,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:43:30,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:31,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:32,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:33,004 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:33,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:34,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:35,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:36,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:37,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:38,005 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:38,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:39,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:40,830 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:41,839 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:42,836 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:43,011 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:43,836 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:44,625 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:43:44,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:44,920 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:43:44,920 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:43:45,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:46,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:43:46,825 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:43:46,826 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:43:46,828 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:43:47,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:47,596 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:43:47,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:48,597 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:43:48,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:48,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:49,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:50,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:51,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:52,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:53,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:53,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:54,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:55,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:56,971 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:57,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:58,866 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:43:58,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:43:59,917 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:43:59,918 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:43:59,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:00,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:01,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:02,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:03,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:04,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:04,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:05,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:06,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:07,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:08,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:09,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:09,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:10,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:11,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:12,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:13,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:14,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:14,633 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:44:14,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:44:14,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:44:14,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:15,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:16,980 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:17,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:18,981 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:19,044 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:19,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:20,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:44:20,127 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:44:20,127 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:44:20,128 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:44:20,620 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:44:20,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:21,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:22,622 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:44:22,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:23,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:24,166 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:24,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:25,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:26,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:27,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:28,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:29,167 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:29,921 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:44:29,922 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:44:29,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:30,982 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:31,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:33,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:34,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:35,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:35,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:36,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:37,099 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:38,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:39,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:40,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:40,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:41,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:42,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:43,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:44,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:44,632 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:44:44,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:44:44,935 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:44:45,108 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:46,014 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:46,105 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:47,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:48,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:49,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:50,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:50,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:44:50,360 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:44:50,360 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:44:50,366 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:44:50,645 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:44:51,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:51,399 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:52,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:52,647 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:44:53,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:54,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:55,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:56,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:56,400 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:44:57,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:58,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:59,111 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:44:59,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:44:59,925 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:45:00,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:01,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:02,038 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:02,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:03,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:04,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:05,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:06,186 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:07,038 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:07,186 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:08,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:09,186 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:10,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:11,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:12,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:12,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:13,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:14,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:14,635 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:45:14,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:45:14,925 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:45:15,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:16,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:17,090 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:17,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:18,193 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:19,194 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:20,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:21,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:21,365 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:45:21,366 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:45:21,367 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:45:21,369 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:45:21,675 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:45:22,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:22,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:22,676 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:45:23,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:24,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:25,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:26,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:27,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:27,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:28,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:29,195 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:29,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:45:29,925 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:45:30,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:31,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:32,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:32,995 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:33,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:34,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:35,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:36,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:37,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:37,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:38,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:39,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:40,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:41,196 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:42,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:42,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:43,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:44,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:44,639 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:45:44,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:45:44,925 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:45:45,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:46,197 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:47,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:48,089 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:48,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:49,204 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:50,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:51,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:52,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:52,446 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:45:52,447 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:45:52,448 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:45:52,453 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:45:52,703 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:45:53,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:53,489 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:54,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:54,706 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:45:55,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:56,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:57,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:58,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:58,490 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:45:59,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:45:59,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:45:59,925 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:46:00,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:01,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:02,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:03,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:04,084 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:04,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:05,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:06,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:07,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:08,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:09,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:09,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:10,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:11,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:12,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:13,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:14,085 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:14,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:14,643 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:46:14,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:46:14,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:46:15,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:16,212 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:17,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:18,219 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:19,215 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:20,076 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:20,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:21,198 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:46:21,200 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:46:21,201 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:46:21,202 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:46:21,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:21,728 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:46:22,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:22,729 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:46:23,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:24,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:25,241 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:25,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:26,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:27,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:28,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:29,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:29,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:46:29,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:46:30,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:31,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:31,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:32,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:33,673 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:34,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:35,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:36,019 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:36,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:37,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:38,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:39,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:40,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:41,020 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:41,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:42,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:43,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:44,647 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:46:44,670 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:44,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:46:44,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:46:45,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:46,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:46,676 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:47,678 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:48,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:49,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:50,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:51,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:46:51,032 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:46:51,032 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:46:51,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:51,034 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:46:51,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:51,756 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:46:52,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:52,757 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:46:53,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:54,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:55,679 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:56,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:46:56,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:57,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:58,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:59,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:46:59,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:46:59,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:47:00,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:01,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:02,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:02,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:03,680 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:04,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:05,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:06,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:07,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:07,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:08,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:09,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:10,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:11,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:12,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:12,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:13,681 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:14,651 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:47:14,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:14,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:47:14,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:47:15,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:16,682 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:17,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:17,689 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:18,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:19,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:20,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:21,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:22,097 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:22,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:22,783 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:47:22,785 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:47:22,785 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:47:22,787 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:47:22,793 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:47:23,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:24,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:24,795 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:47:25,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:26,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:27,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:27,825 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:28,686 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:29,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:29,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:47:29,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:47:30,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:31,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:32,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:33,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:33,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:34,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:35,687 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:36,816 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:37,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:38,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:38,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:39,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:40,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:41,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:42,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:43,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:43,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:44,655 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:47:44,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:44,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:47:44,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:47:45,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:46,730 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:47,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:48,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:48,996 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:49,731 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:50,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:51,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:52,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:53,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:54,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:54,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:55,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:56,247 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:47:56,248 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:47:56,249 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:47:56,251 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:47:56,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:56,818 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:47:57,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:58,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:58,819 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:47:59,290 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:47:59,738 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:47:59,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:47:59,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:48:00,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:01,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:02,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:03,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:04,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:05,015 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:05,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:06,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:07,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:08,739 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:09,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:10,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:10,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:11,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:12,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:13,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:14,659 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:48:14,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:14,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:48:14,926 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:48:15,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:15,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:16,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:17,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:18,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:19,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:20,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:20,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:21,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:22,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:23,743 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:24,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:25,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:25,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:26,747 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:27,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:28,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:48:28,033 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:48:28,033 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:48:28,034 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:48:28,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:28,843 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:48:28,843 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:48:29,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:29,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:48:29,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:48:30,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:30,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:31,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:32,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:33,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:34,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:35,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:35,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:36,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:37,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:38,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:39,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:40,102 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:40,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:41,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:42,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:43,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:44,663 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:48:44,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:44,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:48:44,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:48:45,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:46,090 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:46,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:47,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:48,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:49,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:50,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:51,100 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:51,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:52,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:53,763 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:54,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:55,466 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:48:55,467 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:48:55,468 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:48:55,469 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:48:55,867 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:48:55,881 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:56,507 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:48:56,868 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:48:56,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:57,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:58,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:59,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:48:59,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:48:59,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:49:00,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:01,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:02,056 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:02,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:03,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:04,877 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:05,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:06,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:07,057 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:07,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:08,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:09,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:10,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:11,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:12,057 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:12,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:13,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:14,667 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:49:14,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:14,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:49:14,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:49:15,878 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:16,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:17,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:18,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:18,879 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:19,887 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:20,883 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:21,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:22,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:23,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:23,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:24,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:25,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:49:25,624 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:49:25,625 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:49:25,626 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:49:25,890 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:49:26,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:26,891 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:49:27,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:28,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:28,665 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:29,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:29,926 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:49:29,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:49:30,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:31,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:32,039 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:33,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:34,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:34,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:35,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:36,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:37,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:38,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:39,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:39,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:40,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:41,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:42,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:43,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:44,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:44,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:44,671 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:49:44,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:49:44,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:49:45,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:46,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:47,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:48,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:49,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:50,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:50,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:51,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:52,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:53,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:54,045 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:55,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:49:55,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:56,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:57,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:58,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:58,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:49:58,311 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:49:58,312 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:49:58,313 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:49:58,917 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:49:58,918 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:49:59,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:49:59,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:49:59,927 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:50:00,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:00,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:01,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:02,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:03,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:04,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:05,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:05,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:06,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:07,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:08,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:09,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:10,028 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:10,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:11,122 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:12,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:13,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:14,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:14,674 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:50:14,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:50:14,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:50:15,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:15,119 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:16,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:17,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:18,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:19,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:20,055 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:20,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:21,120 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:22,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:23,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:24,124 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:25,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:25,128 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:26,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:27,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:27,688 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:50:27,689 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:50:27,690 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:50:27,691 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:50:27,943 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:50:28,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:28,944 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:50:29,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:29,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:50:29,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:50:30,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:31,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:31,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:32,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:33,125 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:34,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:35,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:36,009 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:36,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:37,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:38,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:39,126 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:40,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:41,010 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:41,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:42,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:43,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:44,129 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:44,678 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:50:44,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:50:44,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:50:45,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:46,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:46,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:47,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:48,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:49,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:50,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:51,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:51,130 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:52,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:53,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:54,134 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:55,139 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:56,101 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:50:56,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:57,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:57,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:50:57,555 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:50:57,555 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:50:57,556 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:50:57,971 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:50:58,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:58,972 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:50:59,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:50:59,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:50:59,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:51:00,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:01,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:02,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:02,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:03,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:04,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:05,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:06,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:07,017 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:07,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:08,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:09,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:10,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:11,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:12,018 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:12,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:13,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:14,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:14,683 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:51:14,927 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:51:14,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:51:15,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:16,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:17,091 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:17,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:18,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:19,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:20,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:21,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:22,091 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:22,137 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:23,138 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:24,138 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:25,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:26,138 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:27,097 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:27,138 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:28,139 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:29,140 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:29,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:51:29,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:51:30,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:30,232 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:51:30,233 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:51:30,234 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:51:30,235 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:51:31,007 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:51:31,008 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:51:31,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:32,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:32,273 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:33,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:34,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:35,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:36,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:37,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:37,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:38,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:39,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:40,141 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:41,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:42,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:42,274 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:43,380 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:44,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:44,686 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:51:44,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:51:44,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:51:45,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:46,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:47,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:48,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:48,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:49,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:50,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:51,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:52,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:53,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:53,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:54,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:55,143 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:56,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:57,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:58,093 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:51:58,151 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:59,148 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:51:59,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:51:59,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:52:00,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:01,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:01,202 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:52:01,203 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:52:01,203 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:52:01,204 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:52:02,036 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:52:02,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:03,037 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:52:03,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:03,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:04,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:05,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:06,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:07,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:08,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:08,243 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:09,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:10,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:11,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:12,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:13,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:13,244 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:14,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:14,690 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:52:14,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:52:14,928 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:52:15,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:16,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:17,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:18,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:18,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:19,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:20,164 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:21,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:22,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:23,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:23,994 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:24,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:25,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:26,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:27,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:28,165 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:29,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:29,166 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:29,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:52:29,932 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:52:30,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:31,169 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:32,170 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:32,422 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:52:32,423 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:52:32,424 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:52:32,425 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:52:33,062 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:52:33,062 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:52:33,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:34,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:34,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:35,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:36,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:37,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:38,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:39,464 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:39,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:40,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:41,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:42,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:43,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:44,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:44,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:44,694 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:52:44,929 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:52:44,930 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:52:45,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:46,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:47,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:48,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:49,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:50,077 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:50,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:51,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:52,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:53,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:54,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:55,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:52:55,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:56,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:57,550 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:58,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:59,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:52:59,935 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:52:59,939 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:53:00,556 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:01,013 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:01,556 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:02,556 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:03,406 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:53:03,407 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:53:03,407 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:53:03,408 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:53:03,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:04,092 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:53:04,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:05,093 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:53:05,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:06,448 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:06,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:07,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:08,885 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:09,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:10,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:11,448 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:11,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:12,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:13,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:14,698 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:53:15,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:53:15,063 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:53:15,103 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:16,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:17,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:17,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:18,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:19,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:20,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:21,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:22,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:22,152 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:23,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:24,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:25,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:26,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:27,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:27,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:28,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:29,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:30,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:53:30,067 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:53:30,110 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:31,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:31,401 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:53:31,402 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:53:31,403 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:53:31,404 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:53:32,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:32,118 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:53:32,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:33,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:33,119 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:53:34,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:35,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:36,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:37,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:37,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:38,069 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:39,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:40,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:41,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:42,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:42,443 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:43,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:44,070 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:44,702 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:53:45,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:53:45,067 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:53:45,107 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:46,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:47,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:48,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:48,183 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:49,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:50,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:51,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:52,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:53,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:53,185 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:54,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:55,071 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:56,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:57,078 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:58,079 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:53:58,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:53:59,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:00,076 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:54:00,080 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:54:00,114 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:01,088 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:02,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:54:02,042 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:54:02,042 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:54:02,044 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:54:02,144 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:54:03,058 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:03,145 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:54:04,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:04,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:05,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:06,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:07,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:08,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:09,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:09,082 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:10,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:11,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:12,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:13,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:14,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:14,083 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:14,705 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:54:15,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:15,072 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:54:15,073 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:54:16,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:17,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:18,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:19,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:19,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:20,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:21,049 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:22,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:23,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:24,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:24,229 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:25,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:26,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:27,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:28,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:29,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:29,230 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:30,050 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:30,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:54:30,073 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:54:31,057 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:32,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:33,063 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:34,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:35,060 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:35,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:36,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:36,262 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:54:36,263 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:54:36,264 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:54:36,266 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:54:37,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:37,171 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:54:37,172 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:54:38,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:39,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:40,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:40,303 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:41,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:42,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:43,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:44,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:44,709 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:54:45,061 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:45,073 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:54:45,073 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:54:46,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:46,155 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:47,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:48,293 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:49,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:50,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:51,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:51,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:52,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:53,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:54,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:55,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:56,157 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:54:56,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:57,184 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:58,192 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:54:59,187 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:00,074 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:55:00,074 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:55:00,188 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:01,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:01,238 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:02,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:03,189 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:03,769 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:55:03,770 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:55:03,771 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:55:03,772 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:55:04,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:04,197 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:55:05,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:05,198 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:55:06,190 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:06,812 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:07,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:08,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:09,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:10,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:11,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:11,813 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:12,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:13,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:14,191 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:14,713 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:55:15,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:55:15,303 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:55:15,343 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:16,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:17,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:17,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:18,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:19,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:20,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:21,206 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:22,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:22,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:23,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:24,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:25,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:26,207 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:27,223 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:27,445 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:28,208 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:29,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:30,209 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:30,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:55:30,302 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:55:31,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:32,211 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:32,765 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:55:32,767 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:55:32,768 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:55:32,768 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:32,770 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:55:33,227 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:55:33,228 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:55:33,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:34,923 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:35,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:36,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:37,808 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:37,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:38,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:39,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:40,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:41,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:42,809 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:42,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:43,924 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:44,717 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:55:44,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:45,302 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:55:45,303 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:55:45,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:46,925 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:48,112 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:48,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:49,015 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:50,015 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:51,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:52,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:53,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:53,437 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:54,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:55,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:56,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:57,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:58,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:55:58,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:55:59,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:00,023 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:00,307 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:56:00,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:56:01,022 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:02,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:03,024 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:03,930 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:56:03,931 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:56:03,931 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:56:03,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:03,932 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:56:04,251 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:56:04,347 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:05,252 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:56:05,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:06,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:07,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:08,344 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:08,972 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:09,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:10,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:11,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:12,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:13,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:13,973 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:14,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:14,721 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:56:15,304 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:56:15,305 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:56:15,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:16,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:17,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:18,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:19,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:19,408 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:20,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:21,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:22,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:23,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:24,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:24,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:25,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:26,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:27,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:28,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:29,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:29,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:30,308 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:56:30,308 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:56:30,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:31,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:32,351 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:33,352 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:34,355 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:34,433 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:35,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:35,599 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:56:35,600 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:56:35,601 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:56:35,602 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:56:36,281 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:56:36,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:37,282 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:56:37,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:38,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:39,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:39,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:40,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:41,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:42,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:43,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:44,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:44,641 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:44,726 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:56:45,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:56:45,309 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:56:45,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:46,356 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:47,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:48,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:49,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:50,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:50,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:51,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:52,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:53,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:54,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:55,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:55,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:56:56,357 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:57,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:58,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:56:59,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:00,308 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:57:00,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:57:00,358 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:00,461 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:01,362 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:02,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:03,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:04,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:05,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:05,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:57:05,429 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:57:05,430 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:57:05,432 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:57:05,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:06,308 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:57:06,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:07,309 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:57:07,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:08,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:09,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:10,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:10,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:11,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:12,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:13,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:14,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:14,729 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:57:15,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:57:15,309 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:57:15,370 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:16,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:16,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:17,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:18,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:19,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:20,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:21,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:21,448 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:22,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:23,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:24,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:25,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:26,371 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:26,448 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:27,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:28,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:29,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:30,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:57:30,309 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:57:30,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:31,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:32,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:32,390 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:33,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:34,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:35,381 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:36,373 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:37,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:37,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:38,380 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:39,374 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:40,372 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:57:40,374 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:57:40,374 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:57:40,376 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:57:40,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:41,336 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:57:41,337 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:57:41,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:42,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:42,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:43,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:44,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:44,733 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:57:45,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:57:45,309 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:57:45,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:46,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:47,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:47,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:48,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:49,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:50,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:51,377 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:52,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:52,453 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:53,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:54,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:55,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:56,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:57,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:57,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:57:58,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:57:59,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:00,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:58:00,309 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:58:00,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:01,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:02,378 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:02,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:03,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:04,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:05,379 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:06,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:07,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:07,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:08,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:09,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:10,394 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:11,393 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:12,210 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:58:12,211 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:58:12,212 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:58:12,214 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:58:12,367 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:58:12,618 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:13,251 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:13,368 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:58:13,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:14,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:14,738 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:58:15,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:58:15,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:58:15,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:16,614 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:17,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:18,423 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:18,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:19,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:20,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:21,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:22,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:23,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:23,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:24,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:25,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:26,615 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:27,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:28,424 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:28,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:29,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:30,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:58:30,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:58:30,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:31,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:32,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:33,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:34,413 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:34,616 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:35,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:36,623 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:37,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:38,624 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:39,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:39,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:40,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:40,685 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:58:40,686 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:58:40,687 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:58:40,688 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:58:41,391 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:58:41,392 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:58:41,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:42,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:43,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:44,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:44,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:44,742 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:58:45,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:58:45,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:58:45,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:46,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:47,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:48,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:49,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:50,427 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:50,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:51,625 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:52,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:53,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:54,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:55,428 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:58:55,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:56,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:57,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:58,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:58:59,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:00,309 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:59:00,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:59:00,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:01,135 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:01,626 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:02,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:03,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:04,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:05,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:06,136 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:06,627 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:07,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:08,629 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:09,641 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:10,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:11,142 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:11,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:12,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:13,156 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:59:13,157 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:59:13,158 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:59:13,160 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:59:13,420 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:59:13,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:14,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:14,746 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:59:15,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:59:15,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:59:15,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:16,422 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:59:16,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:16,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:17,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:18,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:19,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:20,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:21,430 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:21,635 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:22,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:23,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:24,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:25,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:26,431 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:26,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:27,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:28,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:29,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:30,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:59:30,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:59:30,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:31,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:32,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:32,636 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:33,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:34,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:35,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:36,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:37,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:37,639 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:38,640 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:39,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:40,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:41,638 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:42,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:42,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 10:59:42,562 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 10:59:42,563 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 10:59:42,564 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 10:59:42,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:43,447 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 10:59:43,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:44,448 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 10:59:44,707 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:44,750 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 10:59:45,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 10:59:45,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 10:59:45,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:46,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:47,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:47,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:48,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:49,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:50,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:51,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:52,418 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:52,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:53,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:54,708 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:55,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:56,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:57,419 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 10:59:57,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:58,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 10:59:59,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:00,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:00:00,310 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:00:00,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:01,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:02,457 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:02,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:03,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:04,709 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:05,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:06,710 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:07,458 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:07,716 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:08,712 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:09,719 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:10,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:11,723 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:12,465 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:12,720 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:13,335 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:00:13,336 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:00:13,337 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:00:13,338 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:00:13,472 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:00:13,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:14,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:14,754 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:00:15,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:00:15,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:00:15,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:16,475 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:00:16,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:17,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:18,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:18,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:19,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:20,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:21,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:22,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:23,414 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:23,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:24,749 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:25,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:26,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:27,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:28,415 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:28,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:29,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:30,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:00:30,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:00:30,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:31,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:32,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:33,425 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:33,750 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:34,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:35,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:36,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:37,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:38,426 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:38,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:39,758 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:40,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:41,754 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:42,755 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:43,436 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:43,758 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:44,613 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:00:44,614 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:00:44,614 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:00:44,616 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:00:44,758 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:00:45,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:45,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:00:45,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:00:45,500 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:00:45,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:46,501 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:00:46,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:47,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:48,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:49,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:49,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:50,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:51,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:52,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:53,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:54,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:54,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:55,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:56,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:57,762 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:58,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:00:59,397 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:00:59,803 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:00,310 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:01:00,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:01:00,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:01,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:02,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:03,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:04,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:04,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:05,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:06,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:07,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:08,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:09,447 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:09,804 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:10,811 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:11,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:12,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:13,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:14,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:14,762 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:01:14,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:15,317 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:01:15,317 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:01:15,806 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:16,566 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:01:16,568 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:01:16,568 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:01:16,570 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:01:16,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:17,530 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:01:17,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:18,531 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:01:18,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:19,609 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:19,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:20,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:21,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:22,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:23,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:24,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:24,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:25,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:26,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:27,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:28,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:29,610 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:29,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:30,311 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:01:30,311 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:01:30,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:31,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:32,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:33,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:34,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:35,404 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:35,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:36,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:37,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:38,989 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:39,990 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:40,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:40,993 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:41,999 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:43,000 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:44,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:44,768 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:01:45,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:45,313 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:01:45,317 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:01:46,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:46,217 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:01:46,219 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:01:46,220 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:01:46,220 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:46,222 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:01:46,562 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:01:47,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:48,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:48,564 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:01:49,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:50,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:51,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:51,259 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:52,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:53,001 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:54,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:55,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:56,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:56,260 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:01:57,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:58,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:01:59,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:00,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:00,314 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:02:00,314 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:02:01,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:01,384 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:02,002 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:03,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:04,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:05,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:06,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:06,385 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:07,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:08,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:09,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:10,003 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:11,008 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:11,396 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:12,005 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:13,008 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:14,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:14,773 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:02:15,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:15,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:02:15,322 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:02:16,006 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:16,434 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:16,748 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:02:16,750 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:02:16,750 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:02:16,752 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:02:17,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:17,588 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:02:18,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:18,589 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:02:19,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:20,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:21,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:21,790 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:22,162 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:23,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:24,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:25,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:26,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:26,791 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:27,163 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:28,341 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:29,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:30,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:30,318 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:02:30,319 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:02:31,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:32,248 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:32,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:33,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:34,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:35,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:36,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:37,249 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:37,462 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:38,261 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:39,254 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:40,258 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:41,254 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:42,254 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:42,474 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:43,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:44,094 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:02:44,095 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:02:44,095 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:02:44,097 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:02:44,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:44,612 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:02:44,776 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:02:45,319 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:02:45,319 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:02:45,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:46,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:46,614 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:02:47,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:48,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:48,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:49,504 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:50,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:51,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:52,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:53,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:53,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:54,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:55,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:56,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:57,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:58,387 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:02:58,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:02:59,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:00,319 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:03:00,319 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:03:00,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:01,505 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:02,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:03,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:03,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:04,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:05,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:06,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:07,506 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:08,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:08,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:09,522 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:10,512 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:11,510 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:12,513 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:13,454 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:13,514 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:14,324 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:03:14,325 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:03:14,326 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:03:14,327 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:03:14,635 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:03:14,744 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:14,780 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:03:15,319 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:03:15,319 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:03:15,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:16,636 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:03:16,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:17,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:18,493 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:18,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:19,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:20,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:21,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:22,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:23,494 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:23,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:24,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:25,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:26,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:27,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:28,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:28,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:29,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:30,319 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:03:30,320 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:03:30,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:31,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:32,741 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:33,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:34,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:34,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:35,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:36,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:37,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:38,742 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:39,452 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:39,751 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:40,753 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:41,761 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:42,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:43,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:44,459 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:44,752 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:44,789 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:03:45,322 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:03:45,323 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:03:45,598 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:03:45,599 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:03:45,600 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:03:45,601 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:03:45,659 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:03:46,021 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:47,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:48,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:48,660 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:03:49,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:49,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:50,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:51,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:52,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:53,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:54,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:54,603 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:03:55,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:56,025 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:57,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:58,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:59,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:03:59,604 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:00,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:00,495 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:04:00,496 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:04:01,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:02,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:03,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:04,026 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:04,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:05,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:06,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:07,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:08,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:09,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:09,632 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:10,027 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:11,036 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:12,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:13,030 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:14,037 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:14,634 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:14,790 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:04:15,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:15,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:04:15,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:04:16,031 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:16,226 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:04:16,227 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:04:16,227 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:04:16,230 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:04:16,702 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:04:17,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:18,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:18,703 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:04:19,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:20,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:20,267 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:21,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:22,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:23,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:24,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:25,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:25,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:26,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:27,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:28,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:29,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:30,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:30,268 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:30,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:04:30,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:04:31,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:32,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:33,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:34,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:35,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:35,548 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:36,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:37,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:38,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:39,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:40,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:40,549 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:41,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:42,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:43,035 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:44,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:44,803 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:04:45,044 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:45,407 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:04:45,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:04:45,569 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:46,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:47,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:48,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:49,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:49,240 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:04:49,242 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:04:49,243 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:04:49,245 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:04:49,731 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:04:50,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:50,731 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:04:51,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:51,282 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:52,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:53,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:54,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:55,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:56,040 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:56,282 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:04:57,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:58,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:04:59,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:00,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:00,408 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:05:00,408 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:05:01,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:01,508 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:02,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:03,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:04,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:05,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:06,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:06,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:07,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:08,041 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:09,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:10,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:11,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:11,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:12,042 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:13,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:14,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:14,802 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:05:15,047 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:15,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:05:15,413 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:05:16,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:16,523 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:17,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:18,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:19,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:19,062 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:05:19,063 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:05:19,063 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:05:19,065 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:05:19,772 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:05:20,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:20,773 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:05:21,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:22,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:22,104 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:23,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:24,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:25,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:26,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:27,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:27,104 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:28,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:29,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:30,052 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:30,409 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:05:30,410 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:05:31,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:32,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:32,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:33,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:34,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:35,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:36,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:37,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:37,600 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:38,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:39,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:40,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:41,053 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:42,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:42,601 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:43,054 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:44,787 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:05:44,805 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:05:45,417 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:47,807 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:52,451 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:05:52,452 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:05:52,759 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:05:53,557 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:05:53,701 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:54,669 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:05:58,558 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:01,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:06:02,161 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:06:02,161 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:06:02,631 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:03,585 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:04,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:09,255 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:10,277 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:06:10,969 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:12,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:14,256 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:14,813 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:06:19,257 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:06:19,818 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:19,953 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:06:19,954 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:06:20,381 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:21,345 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:25,032 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:27,565 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:06:27,647 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:28,865 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:30,033 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:35,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:36,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:40,034 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:40,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:06:40,766 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:06:40,766 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:40,767 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:06:41,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:42,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:42,950 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:06:43,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:44,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:44,813 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:06:44,951 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:06:45,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:45,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:46,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:47,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:48,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:49,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:50,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:50,815 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:51,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:06:51,393 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:06:51,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:52,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:53,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:54,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:55,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:56,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:06:56,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:57,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:58,732 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:06:59,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:00,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:01,534 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:01,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:02,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:03,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:04,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:05,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:06,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:07:06,393 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:07:06,733 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:07,469 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:07,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:08,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:09,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:10,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:11,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:12,470 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:12,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:13,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:14,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:14,817 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:07:15,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:16,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:17,734 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:17,820 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:18,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:19,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:20,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:21,392 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:07:21,393 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:07:21,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:22,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:23,553 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:23,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:24,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:25,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:26,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:27,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:28,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:28,735 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:29,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:30,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:31,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:32,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:33,554 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:33,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:38,555 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:43,559 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:44,820 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:07:48,822 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:53,823 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:53,887 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:07:53,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:53,928 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:07:53,929 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:07:54,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:55,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:56,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:57,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:58,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:07:59,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:07:59,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:00,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:01,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:02,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:03,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:04,016 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:04,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:05,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:06,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:07,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:08,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:08:08,891 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:08:08,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:09,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:09,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:10,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:11,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:12,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:13,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:14,043 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:14,824 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:08:14,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:15,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:16,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:17,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:18,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:19,827 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:19,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:20,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:21,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:22,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:23,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:08:23,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:08:23,931 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:24,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:24,998 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:25,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:26,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:27,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:28,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:29,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:29,999 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:30,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:31,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:32,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:33,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:34,894 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:34,999 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:35,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:36,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:37,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:38,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:08:38,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:08:38,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:39,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:40,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:40,970 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:41,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:42,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:43,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:44,829 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:08:44,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:45,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:46,831 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:46,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:47,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:48,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:49,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:50,206 ERROR gpu :287141 [interfaces.py:monitor():142] Failed to sample metric: Not Supported 2025-01-21 11:08:50,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:51,832 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:51,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:52,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:53,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:08:53,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:08:53,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:54,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:55,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:56,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:57,045 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:08:57,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:58,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:08:59,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:00,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:01,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:02,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:02,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:03,897 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:04,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:05,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:06,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:07,046 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:07,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:08,891 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:09:08,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:09:08,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:09,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:10,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:11,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:12,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:12,983 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:13,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:14,833 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:09:14,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:15,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:16,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:17,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:18,835 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:18,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:19,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:20,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:21,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:22,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:23,836 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:23,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:09:23,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:09:23,933 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:24,899 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:25,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:26,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:27,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:28,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:28,987 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:29,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:30,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:31,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:32,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:33,900 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:33,988 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:34,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:35,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:36,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:37,901 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:38,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:09:38,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:09:38,932 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:09:39,979 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:44,837 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:09:45,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:50,840 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:09:55,841 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:00,843 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:05,844 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:10,845 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:14,842 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:10:16,844 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:18,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:10:18,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:18,478 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:10:18,479 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:10:19,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:20,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:21,367 DEBUG SenderThread:287141 [sender.py:send():379] send: telemetry 2025-01-21 11:10:21,368 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,369 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: partial_history 2025-01-21 11:10:21,371 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,372 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,373 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,374 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,375 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,375 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,377 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,377 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,378 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,379 DEBUG SenderThread:287141 [sender.py:send():379] send: history 2025-01-21 11:10:21,379 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: summary_record 2025-01-21 11:10:21,380 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:10:21,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:22,232 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:10:22,381 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:22,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:23,233 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:10:23,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:24,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:25,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:26,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:27,386 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:27,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:28,236 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/config.yaml 2025-01-21 11:10:28,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:29,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:30,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:31,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:32,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:32,707 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:33,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:33,442 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:10:33,443 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:10:34,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:35,439 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:36,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:37,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:38,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:38,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:39,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:40,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:41,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:42,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:43,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:43,560 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:44,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:44,845 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:10:45,440 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:47,246 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:10:48,346 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:48,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:53,958 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:10:55,174 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:10:55,611 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:10:55,611 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:10:55,678 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:57,286 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:10:59,745 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:03,943 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:11:04,746 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:04,829 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:05,536 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:11:05,536 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:11:06,058 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:10,683 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:12,667 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:11:13,479 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:14,509 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:14,849 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:11:15,780 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:15,851 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:20,852 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:23,637 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:11:23,637 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:11:23,907 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:25,306 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:26,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:31,438 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:11:31,717 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:32,405 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:34,483 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:36,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:41,051 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: keepalive 2025-01-21 11:11:41,718 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:44,853 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:11:45,895 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:45,896 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: stop_status 2025-01-21 11:11:45,896 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: stop_status 2025-01-21 11:11:45,936 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:46,885 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:11:46,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:47,045 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:47,898 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:48,064 DEBUG SenderThread:287141 [sender.py:send():379] send: exit 2025-01-21 11:11:48,064 INFO SenderThread:287141 [sender.py:send_exit():586] handling exit code: 0 2025-01-21 11:11:48,064 INFO SenderThread:287141 [sender.py:send_exit():588] handling runtime: 9514 2025-01-21 11:11:48,066 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:11:48,066 INFO SenderThread:287141 [sender.py:send_exit():594] send defer 2025-01-21 11:11:48,067 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:48,067 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 0 2025-01-21 11:11:48,067 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:48,067 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 0 2025-01-21 11:11:48,067 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 1 2025-01-21 11:11:48,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:48,068 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 1 2025-01-21 11:11:48,068 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:48,068 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 1 2025-01-21 11:11:48,068 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 2 2025-01-21 11:11:48,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:48,068 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 2 2025-01-21 11:11:48,068 INFO HandlerThread:287141 [system_monitor.py:finish():203] Stopping system monitor 2025-01-21 11:11:48,069 DEBUG SystemMonitor:287141 [system_monitor.py:_start():179] Finished system metrics aggregation loop 2025-01-21 11:11:48,069 DEBUG SystemMonitor:287141 [system_monitor.py:_start():183] Publishing last batch of metrics 2025-01-21 11:11:48,071 INFO HandlerThread:287141 [interfaces.py:finish():200] Joined cpu monitor 2025-01-21 11:11:48,071 INFO HandlerThread:287141 [interfaces.py:finish():200] Joined disk monitor 2025-01-21 11:11:48,887 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:11:48,887 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:11:49,884 INFO HandlerThread:287141 [interfaces.py:finish():200] Joined gpu monitor 2025-01-21 11:11:49,884 INFO HandlerThread:287141 [interfaces.py:finish():200] Joined memory monitor 2025-01-21 11:11:49,884 INFO HandlerThread:287141 [interfaces.py:finish():200] Joined network monitor 2025-01-21 11:11:49,884 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:49,885 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,885 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 2 2025-01-21 11:11:49,885 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 3 2025-01-21 11:11:49,886 DEBUG SenderThread:287141 [sender.py:send():379] send: stats 2025-01-21 11:11:49,886 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:49,886 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:49,886 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 3 2025-01-21 11:11:49,887 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,887 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 3 2025-01-21 11:11:49,887 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 4 2025-01-21 11:11:49,888 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:49,888 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 4 2025-01-21 11:11:49,888 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,889 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 4 2025-01-21 11:11:49,889 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 5 2025-01-21 11:11:49,889 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:49,889 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 5 2025-01-21 11:11:49,890 DEBUG SenderThread:287141 [sender.py:send():379] send: summary 2025-01-21 11:11:49,891 INFO SenderThread:287141 [sender.py:_save_file():1454] saving file wandb-summary.json with policy end 2025-01-21 11:11:49,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,892 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 5 2025-01-21 11:11:49,892 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 6 2025-01-21 11:11:49,892 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:49,892 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 6 2025-01-21 11:11:49,892 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,892 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 6 2025-01-21 11:11:49,893 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 7 2025-01-21 11:11:49,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: status_report 2025-01-21 11:11:49,893 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:49,893 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 7 2025-01-21 11:11:49,893 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:49,893 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 7 2025-01-21 11:11:50,064 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:50,889 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:11:52,047 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 8 2025-01-21 11:11:52,047 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:52,048 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:52,048 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 8 2025-01-21 11:11:52,048 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:52,048 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 8 2025-01-21 11:11:52,049 INFO SenderThread:287141 [job_builder.py:build():440] Attempting to build job artifact 2025-01-21 11:11:52,052 INFO SenderThread:287141 [job_builder.py:_get_source_type():569] is repo sourced job 2025-01-21 11:11:52,066 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:52,359 INFO SenderThread:287141 [job_builder.py:build():545] adding wandb-job metadata file 2025-01-21 11:11:52,374 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 9 2025-01-21 11:11:52,375 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:52,375 DEBUG SenderThread:287141 [sender.py:send():379] send: artifact 2025-01-21 11:11:52,380 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:52,380 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 9 2025-01-21 11:11:52,891 INFO Thread-12 :287141 [dir_watcher.py:_on_file_modified():288] file/dir modified: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:11:52,953 INFO wandb-upload_1:287141 [upload_job.py:push():85] Skipped uploading /home/ctpham_umass_edu/.local/share/wandb/artifacts/staging/tmpdsou_5sx 2025-01-21 11:11:53,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:53,174 INFO wandb-upload_0:287141 [upload_job.py:push():88] Uploaded file /tmp/tmpva4miryq/wandb-job.json 2025-01-21 11:11:53,852 INFO SenderThread:287141 [sender.py:send_artifact():1530] sent artifact job-https___github.com_chtmp223_BookGen-dev.git_prolong-final_finetune.py - {'id': 'QXJ0aWZhY3Q6MTQ1NjUxMTgyMA==', 'state': 'PENDING', 'artifactSequence': {'id': 'QXJ0aWZhY3RDb2xsZWN0aW9uOjQ5NzE2NzEyNQ==', 'latestArtifact': {'id': 'QXJ0aWZhY3Q6MTQ1MjE3OTAwMA==', 'versionIndex': 29}}} 2025-01-21 11:11:53,852 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:53,852 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 9 2025-01-21 11:11:53,852 INFO SenderThread:287141 [dir_watcher.py:finish():358] shutting down directory watcher 2025-01-21 11:11:53,892 INFO SenderThread:287141 [dir_watcher.py:finish():388] scan: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files 2025-01-21 11:11:53,893 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-metadata.json wandb-metadata.json 2025-01-21 11:11:53,893 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/config.yaml config.yaml 2025-01-21 11:11:53,893 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json wandb-summary.json 2025-01-21 11:11:53,893 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/requirements.txt requirements.txt 2025-01-21 11:11:53,897 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/conda-environment.yaml conda-environment.yaml 2025-01-21 11:11:53,898 INFO SenderThread:287141 [dir_watcher.py:finish():402] scan save: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log output.log 2025-01-21 11:11:53,910 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 10 2025-01-21 11:11:53,914 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:53,914 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:53,914 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 10 2025-01-21 11:11:53,915 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:53,915 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 10 2025-01-21 11:11:53,915 INFO SenderThread:287141 [file_pusher.py:finish():169] shutting down file pusher 2025-01-21 11:11:54,068 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:54,069 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:54,100 INFO wandb-upload_1:287141 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/config.yaml 2025-01-21 11:11:54,163 INFO wandb-upload_2:287141 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/wandb-summary.json 2025-01-21 11:11:54,163 INFO wandb-upload_0:287141 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/requirements.txt 2025-01-21 11:11:54,174 INFO wandb-upload_4:287141 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/output.log 2025-01-21 11:11:54,273 INFO wandb-upload_3:287141 [upload_job.py:push():130] Uploaded file /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/files/conda-environment.yaml 2025-01-21 11:11:54,474 INFO Thread-11 (_thread_body):287141 [sender.py:transition_state():614] send defer: 11 2025-01-21 11:11:54,475 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:54,475 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 11 2025-01-21 11:11:54,475 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:54,476 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 11 2025-01-21 11:11:54,476 INFO SenderThread:287141 [file_pusher.py:join():175] waiting for file pusher 2025-01-21 11:11:54,476 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 12 2025-01-21 11:11:54,476 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:54,476 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 12 2025-01-21 11:11:54,477 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:54,477 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 12 2025-01-21 11:11:54,477 INFO SenderThread:287141 [file_stream.py:finish():601] file stream finish called 2025-01-21 11:11:54,725 INFO SenderThread:287141 [file_stream.py:finish():605] file stream finish is done 2025-01-21 11:11:54,725 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 13 2025-01-21 11:11:54,726 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:54,726 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 13 2025-01-21 11:11:54,726 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:54,726 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 13 2025-01-21 11:11:54,726 INFO SenderThread:287141 [sender.py:transition_state():614] send defer: 14 2025-01-21 11:11:54,727 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: defer 2025-01-21 11:11:54,727 INFO HandlerThread:287141 [handler.py:handle_request_defer():184] handle defer: 14 2025-01-21 11:11:54,727 DEBUG SenderThread:287141 [sender.py:send():379] send: final 2025-01-21 11:11:54,727 DEBUG SenderThread:287141 [sender.py:send():379] send: footer 2025-01-21 11:11:54,728 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: defer 2025-01-21 11:11:54,728 INFO SenderThread:287141 [sender.py:send_request_defer():610] handle sender defer: 14 2025-01-21 11:11:54,728 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:54,729 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:54,729 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: server_info 2025-01-21 11:11:54,730 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: server_info 2025-01-21 11:11:54,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: poll_exit 2025-01-21 11:11:54,736 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: get_summary 2025-01-21 11:11:54,737 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: sampled_history 2025-01-21 11:11:54,740 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: internal_messages 2025-01-21 11:11:54,790 DEBUG SenderThread:287141 [sender.py:send_request():406] send_request: poll_exit 2025-01-21 11:11:54,791 INFO MainThread:287141 [wandb_run.py:_footer_history_summary_info():3991] rendering history 2025-01-21 11:11:54,792 INFO MainThread:287141 [wandb_run.py:_footer_history_summary_info():4023] rendering summary 2025-01-21 11:11:54,793 INFO MainThread:287141 [wandb_run.py:_footer_sync_info():3950] logging synced files 2025-01-21 11:11:54,794 DEBUG HandlerThread:287141 [handler.py:handle_request():158] handle_request: shutdown 2025-01-21 11:11:54,794 INFO HandlerThread:287141 [handler.py:finish():882] shutting down handler 2025-01-21 11:11:55,736 INFO WriterThread:287141 [datastore.py:close():296] close: /scratch3/workspace/ctpham_umass_edu-ft/_llama-3.1-8b-instruct_bsz-16_lr-1e-6_epochs-1_/wandb/run-20250121_083313-ma4iz8vp/run-ma4iz8vp.wandb 2025-01-21 11:11:55,791 INFO SenderThread:287141 [sender.py:finish():1608] shutting down sender 2025-01-21 11:11:55,791 INFO SenderThread:287141 [file_pusher.py:finish():169] shutting down file pusher 2025-01-21 11:11:55,791 INFO SenderThread:287141 [file_pusher.py:join():175] waiting for file pusher