File size: 8,397 Bytes
d787f60 2c4c5b3 |
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 |
2021-09-20 14:28:10,787 INFO MainThread:8660 [wandb_setup.py:_flush():69] setting env: {}
2021-09-20 14:28:10,787 INFO MainThread:8660 [wandb_setup.py:_flush():69] setting login settings: {}
2021-09-20 14:28:10,787 INFO MainThread:8660 [wandb_init.py:_log_setup():348] Logging user logs to /home/leandro/codeparrot/wandb/run-20210920_142810-36cw69uv/logs/debug.log
2021-09-20 14:28:10,787 INFO MainThread:8660 [wandb_init.py:_log_setup():349] Logging internal logs to /home/leandro/codeparrot/wandb/run-20210920_142810-36cw69uv/logs/debug-internal.log
2021-09-20 14:28:10,788 INFO MainThread:8660 [wandb_init.py:init():381] calling init triggers
2021-09-20 14:28:10,788 INFO MainThread:8660 [wandb_init.py:init():386] wandb.init called with sweep_config: {}
config: {'train_batch_size': 2, 'valid_batch_size': 2, 'weight_decay': 0.1, 'shuffle_buffer': 1000, 'learning_rate': 0.0002, 'lr_scheduler_type': 'cosine', 'num_warmup_steps': 750, 'gradient_accumulation_steps': 16, 'max_train_steps': 50000, 'max_eval_steps': -1, 'seq_length': 1024, 'seed': 1, 'save_checkpoint_steps': 50000, 'backend': 'nccl', 'deepspeed_plugin': 'None', 'distributed_type': 'DistributedType.MULTI_GPU', 'num_processes': '16', 'process_index': '0', 'local_process_index': '0', 'device': 'cuda:0', 'use_fp16': 'True', 'initialized': 'True'}
2021-09-20 14:28:10,788 INFO MainThread:8660 [wandb_init.py:init():430] starting backend
2021-09-20 14:28:10,788 INFO MainThread:8660 [backend.py:_multiprocessing_setup():70] multiprocessing start_methods=fork,spawn,forkserver, using: spawn
2021-09-20 14:28:10,805 INFO MainThread:8660 [backend.py:ensure_launched():135] starting backend process...
2021-09-20 14:28:10,816 INFO MainThread:8660 [backend.py:ensure_launched():139] started backend process with pid: 9038
2021-09-20 14:28:10,818 INFO MainThread:8660 [wandb_init.py:init():435] backend started and connected
2021-09-20 14:28:10,825 INFO MainThread:8660 [wandb_init.py:init():494] updated telemetry
2021-09-20 14:28:10,826 INFO MainThread:8660 [wandb_init.py:init():517] communicating current version
2021-09-20 14:28:11,406 INFO MainThread:8660 [wandb_init.py:init():522] got version response
2021-09-20 14:28:11,406 INFO MainThread:8660 [wandb_init.py:init():530] communicating run to backend with 30 second timeout
2021-09-20 14:28:11,486 INFO MainThread:8660 [wandb_init.py:init():557] starting run threads in backend
2021-09-20 14:28:12,872 INFO MainThread:8660 [wandb_run.py:_console_start():1605] atexit reg
2021-09-20 14:28:12,873 INFO MainThread:8660 [wandb_run.py:_redirect():1479] redirect: SettingsConsole.REDIRECT
2021-09-20 14:28:12,873 INFO MainThread:8660 [wandb_run.py:_redirect():1484] Redirecting console.
2021-09-20 14:28:12,876 INFO MainThread:8660 [wandb_run.py:_redirect():1540] Redirects installed.
2021-09-20 14:28:12,876 INFO MainThread:8660 [wandb_init.py:init():582] run started, returning control to user process
2021-09-28 04:43:42,819 INFO MainThread:8660 [wandb_run.py:_atexit_cleanup():1575] got exitcode: 0
2021-09-28 04:43:42,820 INFO MainThread:8660 [wandb_run.py:_restore():1547] restore
2021-09-28 04:43:45,246 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 1
}
pusher_stats {
uploaded_bytes: 792
total_bytes: 792
}
2021-09-28 04:43:45,595 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 1
}
pusher_stats {
uploaded_bytes: 792
total_bytes: 792
}
2021-09-28 04:43:45,944 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 3
}
pusher_stats {
uploaded_bytes: 792
total_bytes: 4501
}
2021-09-28 04:43:46,045 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 2743423
total_bytes: 148562394
}
2021-09-28 04:43:46,147 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 16759935
total_bytes: 148562394
}
2021-09-28 04:43:46,248 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 27868287
total_bytes: 148562394
}
2021-09-28 04:43:46,349 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 39042175
total_bytes: 148562394
}
2021-09-28 04:43:46,450 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 50535551
total_bytes: 148562394
}
2021-09-28 04:43:46,551 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 61062271
total_bytes: 148562394
}
2021-09-28 04:43:46,652 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 69983359
total_bytes: 148562394
}
2021-09-28 04:43:46,753 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 80895103
total_bytes: 148562394
}
2021-09-28 04:43:46,855 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 90905727
total_bytes: 148562394
}
2021-09-28 04:43:46,957 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 101956735
total_bytes: 148562394
}
2021-09-28 04:43:47,058 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 111901823
total_bytes: 148562394
}
2021-09-28 04:43:47,159 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 122363007
total_bytes: 148562394
}
2021-09-28 04:43:47,260 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 133414015
total_bytes: 148562394
}
2021-09-28 04:43:47,361 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 141810815
total_bytes: 148562394
}
2021-09-28 04:43:47,462 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:47,563 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:47,664 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:47,765 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:47,866 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:47,973 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
2021-09-28 04:43:48,123 INFO MainThread:8660 [wandb_run.py:_wait_for_finish():1697] got exit ret: done: true
exit_result {
}
file_counts {
wandb_count: 6
}
pusher_stats {
uploaded_bytes: 148562394
total_bytes: 148562394
}
local_info {
}
2021-09-28 04:43:49,213 INFO MainThread:8660 [wandb_run.py:_show_summary():1853] rendering summary
2021-09-28 04:43:49,213 INFO MainThread:8660 [wandb_run.py:_show_history():1898] rendering history
2021-09-28 04:43:49,214 INFO MainThread:8660 [wandb_run.py:_show_files():1947] logging synced files
|