2021-09-20 14:28:10,787 INFO    MainThread:8660 [wandb_setup.py:_flush():69] setting env: {}
2021-09-20 14:28:10,787 INFO    MainThread:8660 [wandb_setup.py:_flush():69] setting login settings: {}
2021-09-20 14:28:10,787 INFO    MainThread:8660 [wandb_init.py:_log_setup():348] Logging user logs to /home/leandro/codeparrot/wandb/run-20210920_142810-36cw69uv/logs/debug.log
2021-09-20 14:28:10,787 INFO    MainThread:8660 [wandb_init.py:_log_setup():349] Logging internal logs to /home/leandro/codeparrot/wandb/run-20210920_142810-36cw69uv/logs/debug-internal.log
2021-09-20 14:28:10,788 INFO    MainThread:8660 [wandb_init.py:init():381] calling init triggers
2021-09-20 14:28:10,788 INFO    MainThread:8660 [wandb_init.py:init():386] wandb.init called with sweep_config: {}
config: {'train_batch_size': 2, 'valid_batch_size': 2, 'weight_decay': 0.1, 'shuffle_buffer': 1000, 'learning_rate': 0.0002, 'lr_scheduler_type': 'cosine', 'num_warmup_steps': 750, 'gradient_accumulation_steps': 16, 'max_train_steps': 50000, 'max_eval_steps': -1, 'seq_length': 1024, 'seed': 1, 'save_checkpoint_steps': 50000, 'backend': 'nccl', 'deepspeed_plugin': 'None', 'distributed_type': 'DistributedType.MULTI_GPU', 'num_processes': '16', 'process_index': '0', 'local_process_index': '0', 'device': 'cuda:0', 'use_fp16': 'True', 'initialized': 'True'}
2021-09-20 14:28:10,788 INFO    MainThread:8660 [wandb_init.py:init():430] starting backend
2021-09-20 14:28:10,788 INFO    MainThread:8660 [backend.py:_multiprocessing_setup():70] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2021-09-20 14:28:10,805 INFO    MainThread:8660 [backend.py:ensure_launched():135] starting backend process...
2021-09-20 14:28:10,816 INFO    MainThread:8660 [backend.py:ensure_launched():139] started backend process with pid: 9038
2021-09-20 14:28:10,818 INFO    MainThread:8660 [wandb_init.py:init():435] backend started and connected
2021-09-20 14:28:10,825 INFO    MainThread:8660 [wandb_init.py:init():494] updated telemetry
2021-09-20 14:28:10,826 INFO    MainThread:8660 [wandb_init.py:init():517] communicating current version
2021-09-20 14:28:11,406 INFO    MainThread:8660 [wandb_init.py:init():522] got version response 
2021-09-20 14:28:11,406 INFO    MainThread:8660 [wandb_init.py:init():530] communicating run to backend with 30 second timeout
2021-09-20 14:28:11,486 INFO    MainThread:8660 [wandb_init.py:init():557] starting run threads in backend
2021-09-20 14:28:12,872 INFO    MainThread:8660 [wandb_run.py:_console_start():1605] atexit reg
2021-09-20 14:28:12,873 INFO    MainThread:8660 [wandb_run.py:_redirect():1479] redirect: SettingsConsole.REDIRECT
2021-09-20 14:28:12,873 INFO    MainThread:8660 [wandb_run.py:_redirect():1484] Redirecting console.
2021-09-20 14:28:12,876 INFO    MainThread:8660 [wandb_run.py:_redirect():1540] Redirects installed.
2021-09-20 14:28:12,876 INFO    MainThread:8660 [wandb_init.py:init():582] run started, returning control to user process
2021-09-28 04:43:42,819 INFO    MainThread:8660 [wandb_run.py:_atexit_cleanup():1575] got exitcode: 0
2021-09-28 04:43:42,820 INFO    MainThread:8660 [wandb_run.py:_restore():1547] restore
2021-09-28 04:43:45,246 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 1
}
pusher_stats {
  uploaded_bytes: 792
  total_bytes: 792
}

2021-09-28 04:43:45,595 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 1
}
pusher_stats {
  uploaded_bytes: 792
  total_bytes: 792
}

2021-09-28 04:43:45,944 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 3
}
pusher_stats {
  uploaded_bytes: 792
  total_bytes: 4501
}

2021-09-28 04:43:46,045 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 2743423
  total_bytes: 148562394
}

2021-09-28 04:43:46,147 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 16759935
  total_bytes: 148562394
}

2021-09-28 04:43:46,248 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 27868287
  total_bytes: 148562394
}

2021-09-28 04:43:46,349 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 39042175
  total_bytes: 148562394
}

2021-09-28 04:43:46,450 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 50535551
  total_bytes: 148562394
}

2021-09-28 04:43:46,551 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 61062271
  total_bytes: 148562394
}

2021-09-28 04:43:46,652 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 69983359
  total_bytes: 148562394
}

2021-09-28 04:43:46,753 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 80895103
  total_bytes: 148562394
}

2021-09-28 04:43:46,855 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 90905727
  total_bytes: 148562394
}

2021-09-28 04:43:46,957 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 101956735
  total_bytes: 148562394
}

2021-09-28 04:43:47,058 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 111901823
  total_bytes: 148562394
}

2021-09-28 04:43:47,159 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 122363007
  total_bytes: 148562394
}

2021-09-28 04:43:47,260 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 133414015
  total_bytes: 148562394
}

2021-09-28 04:43:47,361 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 141810815
  total_bytes: 148562394
}

2021-09-28 04:43:47,462 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:47,563 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:47,664 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:47,765 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:47,866 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:47,973 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}

2021-09-28 04:43:48,123 INFO    MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: done: true
exit_result {
}
file_counts {
  wandb_count: 6
}
pusher_stats {
  uploaded_bytes: 148562394
  total_bytes: 148562394
}
local_info {
}

2021-09-28 04:43:49,213 INFO    MainThread:8660 [wandb_run.py:_show_summary():1853] rendering summary
2021-09-28 04:43:49,213 INFO    MainThread:8660 [wandb_run.py:_show_history():1898] rendering history
2021-09-28 04:43:49,214 INFO    MainThread:8660 [wandb_run.py:_show_files():1947] logging synced files