Model parameters: d_model 768 ffw_size 3072 kv_size 64 n_heads 12 n_layers 15
Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 15 --hidden-size 768 --num-attention-heads 12 --kv-channels 64 --ffn-hidden-size 3072 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 4 --global-batch-size 256 --train-samples 44_416_143 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-146m91b400m --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 44_416_143 --lr-warmup-samples 444_161 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 100 --save-interval 10000 --eval-interval 10000 --eval-iters 1 --tensorboard-dir tensorboard_146m91b400m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_146m91b400m --load checkpoints_146m91b400m --train-weighted-split-paths-path train400m.txt --valid-weighted-split-paths-path val.txt --data-impl mmap --deepspeed --deepspeed_config ds_configs/3326737.json --zero-stage 0
START 3326737: Thu 16 Mar 2023 11:22:41 PM EET
0: 
0: 
0: ======================= ROCm System Management Interface =======================
0: ================================= Concise Info =================================
0: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
0: 0    43.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 1    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 2    39.0c  93.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 3    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 4    42.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 5    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: 6    39.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
0: 7    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
0: ================================================================================
0: ============================= End of ROCm SMI Log ==============================
1: 
1: 
1: ======================= ROCm System Management Interface =======================
1: ================================= Concise Info =================================
1: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
1: 0    49.0c  87.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 1    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 2    42.0c  92.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 3    49.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 4    44.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 5    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: 6    41.0c  83.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
1: 7    43.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
1: ================================================================================
1: ============================= End of ROCm SMI Log ==============================
7: 
7: 
7: ======================= ROCm System Management Interface =======================
7: ================================= Concise Info =================================
7: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
7: 0    40.0c  88.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 1    52.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 2    46.0c  93.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 3    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 4    43.0c  84.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 5    47.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: 6    40.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
7: 7    50.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
7: ================================================================================
7: ============================= End of ROCm SMI Log ==============================
2: 
2: 
2: ======================= ROCm System Management Interface =======================
2: ================================= Concise Info =================================
2: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
2: 0    43.0c  93.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 1    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 2    45.0c  83.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 3    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 4    40.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 5    42.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: 6    38.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
2: 7    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
2: ================================================================================
2: ============================= End of ROCm SMI Log ==============================
3: 
3: 
3: ======================= ROCm System Management Interface =======================
3: ================================= Concise Info =================================
3: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
3: 0    41.0c  95.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 1    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 2    42.0c  94.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 3    43.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 4    43.0c  93.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 5    50.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: 6    42.0c  96.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
3: 7    41.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
3: ================================================================================
3: ============================= End of ROCm SMI Log ==============================
6: 
6: 
6: ======================= ROCm System Management Interface =======================
6: ================================= Concise Info =================================
6: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
6: 0    45.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 1    44.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 2    38.0c  95.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 3    43.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 4    38.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 5    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: 6    41.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
6: 7    41.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
6: ================================================================================
6: ============================= End of ROCm SMI Log ==============================
5: 
5: 
5: ======================= ROCm System Management Interface =======================
5: ================================= Concise Info =================================
5: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
5: 0    46.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 1    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 2    36.0c  96.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 3    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 4    39.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 5    48.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: 6    37.0c  85.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
5: 7    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
5: ================================================================================
5: ============================= End of ROCm SMI Log ==============================
4: 
4: 
4: ======================= ROCm System Management Interface =======================
4: ================================= Concise Info =================================
4: GPU  Temp   AvgPwr  SCLK    MCLK     Fan  Perf  PwrCap  VRAM%  GPU%  
4: 0    45.0c  89.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 1    42.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 2    40.0c  87.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 3    43.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 4    41.0c  91.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 5    46.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: 6    40.0c  90.0W   800Mhz  1600Mhz  0%   auto  560.0W    0%   0%    
4: 7    45.0c  N/A     800Mhz  1600Mhz  0%   auto  0.0W      0%   0%    
4: ================================================================================
4: ============================= End of ROCm SMI Log ==============================
3: Launching on nid005623 (3/8), master nid005620 port 9999, GPUs 8, CUDA: True
5: Launching on nid005625 (5/8), master nid005620 port 9999, GPUs 8, CUDA: True
2: Launching on nid005622 (2/8), master nid005620 port 9999, GPUs 8, CUDA: True
4: Launching on nid005624 (4/8), master nid005620 port 9999, GPUs 8, CUDA: True
1: Launching on nid005621 (1/8), master nid005620 port 9999, GPUs 8, CUDA: True
7: Launching on nid005627 (7/8), master nid005620 port 9999, GPUs 8, CUDA: True
6: Launching on nid005626 (6/8), master nid005620 port 9999, GPUs 8, CUDA: True
0: Launching on nid005620 (0/8), master nid005620 port 9999, GPUs 8, CUDA: True
7: > setting tensorboard ...
0: using world size: 64, data-parallel-size: 64, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 
0: accumulate and all-reduce gradients in fp32 for bfloat16 data type.
0: using torch.bfloat16 for parameters ...
0: ------------------------ arguments ------------------------
0:   abort_on_unmet_fused_kernel_constraints ......... False
0:   accumulate_allreduce_grads_in_fp32 .............. True
0:   adam_beta1 ...................................... 0.9
0:   adam_beta2 ...................................... 0.999
0:   adam_eps ........................................ 1e-08
0:   adlr_autoresume ................................. False
0:   adlr_autoresume_interval ........................ 1000
0:   apply_query_key_layer_scaling ................... True
0:   apply_residual_connection_post_layernorm ........ False
0:   attention_dropout ............................... 0.1
0:   attention_softmax_in_fp32 ....................... False
0:   bert_binary_head ................................ True
0:   bert_load ....................................... None
0:   bf16 ............................................ True
0:   bias_dropout_fusion ............................. True
0:   bias_gelu_fusion ................................ True
0:   biencoder_projection_dim ........................ 0
0:   biencoder_shared_query_context_model ............ False
0:   block_data_path ................................. None
0:   checkpoint_activations .......................... True
0:   checkpoint_in_cpu ............................... False
0:   checkpoint_num_layers ........................... 1
0:   clip_grad ....................................... 1.0
0:   codecarbon_dir .................................. None
0:   consumed_train_samples .......................... 0
0:   consumed_train_tokens ........................... 0
0:   consumed_valid_samples .......................... 0
0:   contigious_checkpointing ........................ False
0:   cpu_optimizer ................................... False
0:   cpu_torch_adam .................................. False
0:   curriculum_learning ............................. False
0:   data_impl ....................................... mmap
0:   data_parallel_size .............................. 64
0:   data_path ....................................... None
0:   dataloader_type ................................. single
0:   DDP_impl ........................................ local
0:   decoder_seq_length .............................. None
0:   deepscale ....................................... False
0:   deepscale_config ................................ None
0:   deepspeed ....................................... True
0:   deepspeed_activation_checkpointing .............. False
0:   deepspeed_config ................................ ds_configs/3326737.json
0:   deepspeed_mpi ................................... False
0:   distribute_checkpointed_activations ............. False
0:   distributed_backend ............................. nccl
0:   embed_layernorm ................................. False
0:   embedding_path .................................. None
0:   encoder_seq_length .............................. 2048
0:   eod_mask_loss ................................... False
0:   eval_interval ................................... 10000
0:   eval_iters ...................................... 1
0:   eval_only ....................................... None
0:   evidence_data_path .............................. None
0:   exit_duration_in_mins ........................... None
0:   exit_interval ................................... None
0:   ffn_hidden_size ................................. 3072
0:   finetune ........................................ False
0:   fp16 ............................................ False
0:   fp16_lm_cross_entropy ........................... False
0:   fp32_residual_connection ........................ False
0:   gigaflos_no_embeds .............................. 0
0:   global_batch_size ............................... 256
0:   glu_activation .................................. None
0:   hidden_dropout .................................. 0.1
0:   hidden_size ..................................... 768
0:   hysteresis ...................................... 2
0:   ict_head_size ................................... None
0:   ict_load ........................................ None
0:   img_dim ......................................... 224
0:   indexer_batch_size .............................. 128
0:   indexer_log_interval ............................ 1000
0:   inference ....................................... False
0:   init_method_std ................................. 0.02
0:   init_method_xavier_uniform ...................... False
0:   initial_loss_scale .............................. 4294967296
0:   kill_switch_path ................................ kill-switch-146m91b400m
0:   kv_channels ..................................... 64
0:   layer_norm_fusion ............................... True
0:   layernorm_epsilon ............................... 1e-05
0:   lazy_mpu_init ................................... None
0:   load ............................................ checkpoints_146m91b400m
0:   local_rank ...................................... None
0:   log_batch_size_to_tensorboard ................... True
0:   log_interval .................................... 100
0:   log_learning_rate_to_tensorboard ................ True
0:   log_level ....................................... None
0:   log_level_replica ............................... None
0:   log_loss_scale_to_tensorboard ................... True
0:   log_num_zeros_in_grad ........................... False
0:   log_params_norm ................................. False
0:   log_path ........................................ None
0:   log_timers_to_tensorboard ....................... True
0:   log_validation_ppl_to_tensorboard ............... True
0:   loss_on_targets_only ............................ False
0:   loss_scale ...................................... 12.0
0:   loss_scale_window ............................... 1000
0:   lr .............................................. 0.0002
0:   lr_decay_iters .................................. None
0:   lr_decay_samples ................................ 44416143
0:   lr_decay_style .................................. cosine
0:   lr_decay_tokens ................................. None
0:   lr_warmup_fraction .............................. None
0:   lr_warmup_iters ................................. 0
0:   lr_warmup_samples ............................... 444161
0:   make_vocab_size_divisible_by .................... 128
0:   mask_prob ....................................... 0.15
0:   masked_softmax_fusion ........................... True
0:   max_position_embeddings ......................... 2048
0:   mean_noise_span_length .......................... None
0:   memory_centric_tiled_linear ..................... False
0:   merge_file ...................................... gpt2/merges.txt
0:   micro_batch_size ................................ 4
0:   min_loss_scale .................................. 1.0
0:   min_lr .......................................... 2e-05
0:   mmap_warmup ..................................... False
0:   no_load_optim ................................... None
0:   no_load_rng ..................................... None
0:   no_save_optim ................................... None
0:   no_save_rng ..................................... None
0:   noise_density ................................... None
0:   num_attention_heads ............................. 12
0:   num_channels .................................... 3
0:   num_classes ..................................... 1000
0:   num_layers ...................................... 15
0:   num_layers_per_virtual_pipeline_stage ........... None
0:   num_workers ..................................... 2
0:   onnx_safe ....................................... None
0:   openai_gelu ..................................... False
0:   optimizer ....................................... adam
0:   optimizer_fusion ................................ True
0:   override_lr_scheduler ........................... False
0:   pad_vocab_size_to ............................... None
0:   params_dtype .................................... torch.bfloat16
0:   partition_activations ........................... False
0:   patch_dim ....................................... 16
0:   pipeline_model_parallel_size .................... 1
0:   position_embedding_type ......................... PositionEmbeddingType.absolute
0:   pp_partition_method ............................. None
0:   profile_backward ................................ False
0:   query_in_block_prob ............................. 0.1
0:   rampup_batch_size ............................... None
0:   rank ............................................ 0
0:   remote_device ................................... none
0:   reset_attention_mask ............................ False
0:   reset_position_ids .............................. False
0:   reset_progress .................................. None
0:   retriever_report_topk_accuracies ................ []
0:   retriever_score_scaling ......................... False
0:   retriever_seq_length ............................ 256
0:   reweight_loss_based_on_position_frequency ....... False
0:   sample_rate ..................................... 1.0
0:   save ............................................ checkpoints_146m91b400m
0:   save_interval ................................... 10000
0:   scatter_gather_tensors_in_pipeline .............. True
0:   scattered_embeddings ............................ False
0:   seed ............................................ 1234
0:   seq_length ...................................... 2048
0:   sgd_momentum .................................... 0.9
0:   short_seq_prob .................................. 0.1
0:   skip_train_iteration_range ...................... None
0:   split ........................................... None
0:   split_transformers .............................. False
0:   sync_tp_duplicated_parameters ................... False
0:   synchronize_each_layer .......................... False
0:   tensor_model_parallel_size ...................... 1
0:   tensorboard_dir ................................. tensorboard_146m91b400m
0:   tensorboard_log_interval ........................ 1
0:   tensorboard_queue_size .......................... 5
0:   test_weighted_split_paths ....................... None
0:   test_weighted_split_paths_path .................. None
0:   tile_factor ..................................... 1
0:   titles_data_path ................................ None
0:   tokenizer_name_or_path .......................... None
0:   tokenizer_type .................................. GPT2BPETokenizer
0:   train_iters ..................................... None
0:   train_samples ................................... 44416143
0:   train_tokens .................................... None
0:   train_weighted_split_names ...................... ['train']
0:   train_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document']]
0:   train_weighted_split_paths_path ................. None
0:   train_weighted_split_splits ..................... [['0:1']]
0:   train_weighted_split_weights .................... [['1.0']]
0:   universal_checkpoint ............................ False
0:   use_bnb_optimizer ............................... False
0:   use_checkpoint_lr_scheduler ..................... False
0:   use_contiguous_buffers_in_ddp ................... True
0:   use_cpu_initialization .......................... None
0:   use_one_sent_docs ............................... False
0:   use_pin_memory .................................. False
0:   valid_num_workers ............................... 2
0:   valid_weighted_split_names ...................... ['validation']
0:   valid_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document']]
0:   valid_weighted_split_paths_path ................. None
0:   valid_weighted_split_splits ..................... [['0:1']]
0:   valid_weighted_split_weights .................... [['1.0']]
0:   virtual_pipeline_model_parallel_size ............ None
0:   vocab_extra_ids ................................. 0
0:   vocab_file ...................................... gpt2/vocab.json
0:   weight_decay .................................... 0.1
0:   world_size ...................................... 64
0:   zero_allgather_bucket_size ...................... 0.0
0:   zero_contigious_gradients ....................... False
0:   zero_reduce_bucket_size ......................... 0.0
0:   zero_reduce_scatter ............................. False
0:   zero_stage ...................................... 0
0: -------------------- end of arguments ---------------------
0: setting number of micro-batches to constant 1
0: > building GPT2BPETokenizer tokenizer ...
0:  > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304)
0: DeepSpeed general environment info:
0: torch install path ............... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch']
0: torch version .................... 1.13.0+rocm5.2
0: torch cuda version ............... None
0: torch hip version ................ 5.2.21151-afdc89f8
0: nvcc version ..................... None
0: deepspeed install path ........... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/deepspeed']
0: deepspeed info ................... 0.7.5, unknown, unknown
0: deepspeed wheel compiled w. ...... torch 1.13, hip 5.1
0: **** Git info for Megatron: git_hash=unknown git_branch=unknown ****
0: > initializing torch distributed ...
0: [2023-03-16 23:23:26,632] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl
0: > initializing tensor model parallel with size 1
0: > initializing pipeline model parallel with size 1
0: > setting random seeds to 1234 ...
0: > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234
0: > compiling dataset index builder ...
0: make: Entering directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data'
0: make: Nothing to be done for 'default'.
0: make: Leaving directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data'
0: >>> done with dataset index builder. Compilation time: 0.098 seconds
0: > compiling and loading fused kernels ...
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 87
0: [1/1] c++ scaled_upper_triang_masked_softmax_hip.o scaled_upper_triang_masked_softmax_hip.cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o scaled_upper_triang_masked_softmax_cuda.so
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.cpp [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 63
0: ninja: no work to do.
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda_kernel.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_hip_kernel.hip [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified]
0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified]
0: Total number of unsupported CUDA function calls: 0
0: 
0: 
0: Total number of replaced kernel launches: 67
0: [1/1] c++ layer_norm_cuda.o layer_norm_hip_kernel.cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o fused_mix_prec_layer_norm_cuda.so
0: >>> done with compiling and loading fused kernels. Compilation time: 27.183 seconds
0: time to initialize megatron (seconds): -19.398
0: [after megatron is initialized] datetime: 2023-03-16 23:23:56 
0: building GPT model ...
0: [2023-03-16 23:23:56,739] [INFO] [utils.py:827:see_memory_usage] Before Building Model
0: [2023-03-16 23:23:56,739] [INFO] [utils.py:828:see_memory_usage] MA 0.0 GB         Max_MA 0.0 GB         CA 0.0 GB         Max_CA 0 GB 
0: [2023-03-16 23:23:56,740] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 30.56 GB, percent = 6.1%
0: SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None
0: Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pi
0: pe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=4
0: 6, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63}
0: [2023-03-16 23:23:58,724] [INFO] [module.py:366:_partition_layers] Partitioning pipeline stages with method type:transformer
0: stage=0 layers=22
0:      0: _to_float16
0:      1: EmbeddingPipe
0:      2: <lambda>
0:      3: ParallelTransformerLayerPipe
0:      4: ParallelTransformerLayerPipe
0:      5: ParallelTransformerLayerPipe
0:      6: ParallelTransformerLayerPipe
0:      7: ParallelTransformerLayerPipe
0:      8: ParallelTransformerLayerPipe
0:      9: ParallelTransformerLayerPipe
0:     10: ParallelTransformerLayerPipe
0:     11: ParallelTransformerLayerPipe
0:     12: ParallelTransformerLayerPipe
0:     13: ParallelTransformerLayerPipe
0:     14: ParallelTransformerLayerPipe
0:     15: ParallelTransformerLayerPipe
0:     16: ParallelTransformerLayerPipe
0:     17: ParallelTransformerLayerPipe
0:     18: undo
0:     19: MixedFusedLayerNorm
0:     20: EmbeddingPipe
0:     21: float16_to_fp32
0:   loss: CrossEntropy
0: [2023-03-16 23:23:58,924] [INFO] [utils.py:827:see_memory_usage] After Building Model
0: [2023-03-16 23:23:58,925] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB         Max_MA 0.28 GB         CA 0.29 GB         Max_CA 0 GB 
0: [2023-03-16 23:23:58,925] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 30.58 GB, percent = 6.1%
0: setting training iterations to 173500
0: > learning rate decay style: cosine
0: DeepSpeed is enabled.
0: [2023-03-16 23:23:58,927] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.5, git-hash=unknown, git-branch=unknown
0: [2023-03-16 23:24:11,313] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False
0: [2023-03-16 23:24:11,313] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer
0: [2023-03-16 23:24:11,313] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer
0: [2023-03-16 23:24:11,318] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam
0: [2023-03-16 23:24:11,318] [INFO] [logging.py:68:log_dist] [Rank 0] Creating BF16 optimizer
0: [2023-03-16 23:24:11,433] [INFO] [utils.py:827:see_memory_usage] begin bf16_optimizer
0: [2023-03-16 23:24:11,434] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB         Max_MA 0.29 GB         CA 0.31 GB         Max_CA 0 GB 
0: [2023-03-16 23:24:11,434] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.27 GB, percent = 6.2%
0: ninja: no work to do.
0: Time to load utils op: 0.1557788848876953 seconds
0: Time to load utils op: 0.0006306171417236328 seconds
5: ninja: no work to do.
5: Time to load utils op: 0.1482529640197754 seconds
0: Time to load utils op: 0.2033100128173828 seconds
4: Time to load utils op: 0.31090879440307617 seconds
0: Time to load utils op: 0.20261621475219727 seconds
0: Time to load utils op: 0.2024383544921875 seconds
0: Time to load utils op: 0.2028486728668213 seconds
0: Time to load utils op: 0.20282530784606934 seconds
0: Time to load utils op: 0.20265793800354004 seconds
0: Time to load utils op: 0.20308709144592285 seconds
4: Time to load utils op: 0.20319771766662598 secondsTime to load utils op: 0.20260953903198242 seconds
4: 
4: Time to load utils op: 0.20279526710510254 seconds
4: Time to load utils op: 0.20283126831054688 seconds
4: Time to load utils op: 0.2032482624053955 seconds
4: Time to load utils op: 0.20310139656066895 secondsTime to load utils op: 0.20307230949401855 seconds
4: 
5: Time to load utils op: 0.2039203643798828 seconds
5: Time to load utils op: 0.2041788101196289 seconds
5: Time to load utils op: 0.2047593593597412 secondsTime to load utils op: 0.20498871803283691 seconds
5: 
5: Time to load utils op: 0.2046046257019043 seconds
5: Time to load utils op: 0.20483922958374023 seconds
5: Time to load utils op: 0.2047131061553955 seconds
1: Time to load utils op: 0.21193432807922363 secondsTime to load utils op: 0.21169543266296387 seconds
1: 
1: Time to load utils op: 0.21203017234802246 seconds
1: Time to load utils op: 0.2116107940673828 secondsTime to load utils op: 0.21202492713928223 seconds
1: 
1: Time to load utils op: 0.21146225929260254 secondsTime to load utils op: 0.21199727058410645 seconds
1: Time to load utils op: 0.21163654327392578 seconds
1: 
5: Time to load utils op: 0.0007970333099365234 seconds
3: Time to load utils op: 0.2109355926513672 seconds
3: Time to load utils op: 0.2109384536743164 seconds
3: Time to load utils op: 0.21095490455627441 seconds
3: Time to load utils op: 0.21094822883605957 seconds
3: Time to load utils op: 0.21096062660217285 seconds
3: Time to load utils op: 0.21097397804260254 secondsTime to load utils op: 0.2109682559967041 seconds
3: 
3: Time to load utils op: 0.21097707748413086 seconds
2: Time to load utils op: 0.21094465255737305 seconds
2: Time to load utils op: 0.21105742454528809 seconds
2: Time to load utils op: 0.21092867851257324 secondsTime to load utils op: 0.21097350120544434 seconds
2: 
2: Time to load utils op: 0.2106482982635498 secondsTime to load utils op: 0.21232843399047852 seconds
2: 
2: Time to load utils op: 0.21087932586669922 secondsTime to load utils op: 0.21231889724731445 seconds
2: 
0: Time to load utils op: 0.000537872314453125 seconds
0: Time to load utils op: 0.0004906654357910156 seconds
0: Time to load utils op: 0.00037217140197753906 seconds
0: Time to load utils op: 0.0003657341003417969 seconds
0: Time to load utils op: 0.00042891502380371094 seconds
0: Time to load utils op: 0.0003681182861328125 seconds
6: Time to load utils op: 0.21145391464233398 seconds
6: Time to load utils op: 0.2114582061767578 seconds
6: Time to load utils op: 0.2114849090576172 seconds
6: Time to load utils op: 0.21147680282592773 seconds
6: Time to load utils op: 0.21148681640625 seconds
6: Time to load utils op: 0.21149826049804688 seconds
6: Time to load utils op: 0.2115013599395752 seconds
6: Time to load utils op: 0.21150612831115723 seconds
7: Time to load utils op: 0.21138405799865723 secondsTime to load utils op: 0.21139073371887207 seconds
7: 
7: Time to load utils op: 0.2113969326019287 seconds
7: Time to load utils op: 0.21140480041503906 seconds
7: Time to load utils op: 0.21141839027404785 seconds
7: Time to load utils op: 0.21135878562927246 secondsTime to load utils op: 0.21142268180847168 secondsTime to load utils op: 0.2114248275756836 seconds
7: 
7: 
5: Time to load utils op: 0.0004494190216064453 seconds
5: Time to load utils op: 0.0003256797790527344 seconds
5: Time to load utils op: 0.0003762245178222656 seconds
5: Time to load utils op: 0.0003657341003417969 seconds
5: Time to load utils op: 0.00033926963806152344 seconds
5: Time to load utils op: 0.0003490447998046875 seconds
5: Time to load utils op: 0.00033354759216308594 seconds
4: Time to load utils op: 0.0003876686096191406 seconds
4: Time to load utils op: 0.0005404949188232422 seconds
4: Time to load utils op: 0.0005528926849365234 seconds
4: Time to load utils op: 0.0004012584686279297 seconds
4: Time to load utils op: 0.0004086494445800781 seconds
4: Time to load utils op: 0.0005166530609130859 seconds
4: Time to load utils op: 0.0005276203155517578 seconds
4: Time to load utils op: 0.0005502700805664062 seconds
1: Time to load utils op: 0.0009441375732421875 seconds
1: Time to load utils op: 0.001005411148071289 seconds
1: Time to load utils op: 0.0009744167327880859 seconds
1: Time to load utils op: 0.0012500286102294922 seconds
1: Time to load utils op: 0.0012278556823730469 seconds
1: Time to load utils op: 0.0012767314910888672 secondsTime to load utils op: 0.0013451576232910156 seconds
1: 
1: Time to load utils op: 0.0012879371643066406 seconds
3: Time to load utils op: 0.000942230224609375 seconds
3: Time to load utils op: 0.0008344650268554688 seconds
3: Time to load utils op: 0.0011425018310546875 secondsTime to load utils op: 0.0010991096496582031 secondsTime to load utils op: 0.0010395050048828125 seconds
3: 
3: 
3: Time to load utils op: 0.0010590553283691406 seconds
3: Time to load utils op: 0.001116037368774414 seconds
3: Time to load utils op: 0.001088857650756836 seconds
2: Time to load utils op: 0.0003955364227294922 seconds
2: Time to load utils op: 0.0005395412445068359 seconds
2: Time to load utils op: 0.0005524158477783203 seconds
2: Time to load utils op: 0.0005571842193603516 secondsTime to load utils op: 0.0004334449768066406 seconds
2: 
2: Time to load utils op: 0.0005524158477783203 seconds
2: Time to load utils op: 0.00043272972106933594 seconds
2: Time to load utils op: 0.0004363059997558594 seconds
6: Time to load utils op: 0.0011148452758789062 seconds
7: Time to load utils op: 0.0009963512420654297 seconds
6: Time to load utils op: 0.0012254714965820312 seconds
6: Time to load utils op: 0.0013799667358398438 seconds
7: Time to load utils op: 0.0012049674987792969 seconds
6: Time to load utils op: 0.0013990402221679688 seconds
6: Time to load utils op: 0.001466512680053711 seconds
6: Time to load utils op: 0.0014662742614746094 seconds
6: Time to load utils op: 0.001508951187133789 seconds
7: Time to load utils op: 0.0010874271392822266 seconds
6: Time to load utils op: 0.001497030258178711 seconds
7: Time to load utils op: 0.0011119842529296875 seconds
7: Time to load utils op: 0.0012171268463134766 seconds
7: Time to load utils op: 0.0013575553894042969 seconds
7: Time to load utils op: 0.0013360977172851562 seconds
7: Time to load utils op: 0.0013232231140136719 seconds
0: [2023-03-16 23:24:11,759] [INFO] [utils.py:827:see_memory_usage] before initializing group 0
0: [2023-03-16 23:24:11,760] [INFO] [utils.py:828:see_memory_usage] MA 0.28 GB         Max_MA 0.28 GB         CA 0.31 GB         Max_CA 0 GB 
0: [2023-03-16 23:24:11,760] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:11,876] [INFO] [utils.py:827:see_memory_usage] after initializing group 0
0: [2023-03-16 23:24:11,876] [INFO] [utils.py:828:see_memory_usage] MA 0.62 GB         Max_MA 0.62 GB         CA 0.82 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:11,876] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:11,978] [INFO] [utils.py:827:see_memory_usage] before initializing group 1
0: [2023-03-16 23:24:11,979] [INFO] [utils.py:828:see_memory_usage] MA 0.62 GB         Max_MA 0.62 GB         CA 0.82 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:11,979] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,081] [INFO] [utils.py:827:see_memory_usage] after initializing group 1
0: [2023-03-16 23:24:12,081] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB         Max_MA 0.83 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,082] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,182] [INFO] [utils.py:827:see_memory_usage] before initializing group 2
0: [2023-03-16 23:24:12,182] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB         Max_MA 0.83 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,183] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,285] [INFO] [utils.py:827:see_memory_usage] after initializing group 2
0: [2023-03-16 23:24:12,286] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB         Max_MA 0.83 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,286] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,386] [INFO] [utils.py:827:see_memory_usage] before initialize_optimizer
0: [2023-03-16 23:24:12,387] [INFO] [utils.py:828:see_memory_usage] MA 0.83 GB         Max_MA 0.83 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,387] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,492] [INFO] [utils.py:827:see_memory_usage] end initialize_optimizer
0: [2023-03-16 23:24:12,493] [INFO] [utils.py:828:see_memory_usage] MA 0.85 GB         Max_MA 0.85 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,493] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,594] [INFO] [utils.py:827:see_memory_usage] end bf16_optimizer
0: [2023-03-16 23:24:12,595] [INFO] [utils.py:828:see_memory_usage] MA 0.85 GB         Max_MA 0.85 GB         CA 1.13 GB         Max_CA 1 GB 
0: [2023-03-16 23:24:12,595] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory:  used = 31.41 GB, percent = 6.2%
0: [2023-03-16 23:24:12,595] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam
0: [2023-03-16 23:24:12,595] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler
0: [2023-03-16 23:24:12,595] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = <megatron.learning_rates.AnnealingLR object at 0x1547afdd2d30>
0: [2023-03-16 23:24:12,595] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1007:print] DeepSpeedEngine configuration:
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   activation_checkpointing_config  {
0:     "partition_activations": false, 
0:     "contiguous_memory_optimization": false, 
0:     "cpu_checkpointing": false, 
0:     "number_checkpoints": null, 
0:     "synchronize_checkpoint_boundary": false, 
0:     "profile": false
0: }
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True}
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   amp_enabled .................. False
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   amp_params ................... False
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   autotuning_config ............ {
0:     "enabled": false, 
0:     "start_step": null, 
0:     "end_step": null, 
0:     "metric_path": null, 
0:     "arg_mappings": null, 
0:     "metric": "throughput", 
0:     "model_info": null, 
0:     "results_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_results", 
0:     "exps_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_exps", 
0:     "overwrite": true, 
0:     "fast": true, 
0:     "start_profile_step": 3, 
0:     "end_profile_step": 5, 
0:     "tuner_type": "gridsearch", 
0:     "tuner_early_stopping": 5, 
0:     "tuner_num_trials": 50, 
0:     "model_info_path": null, 
0:     "mp_size": 1, 
0:     "max_train_batch_size": null, 
0:     "min_train_batch_size": 1, 
0:     "max_train_micro_batch_size_per_gpu": 1.024000e+03, 
0:     "min_train_micro_batch_size_per_gpu": 1, 
0:     "num_tuning_micro_batch_sizes": 3
0: }
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   bfloat16_enabled ............. True
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   checkpoint_parallel_write_pipeline  False
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   checkpoint_tag_validation_enabled  True
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   checkpoint_tag_validation_fail  False
0: [2023-03-16 23:24:12,596] [INFO] [config.py:1011:print]   comms_config ................. <deepspeed.comm.config.DeepSpeedCommsConfig object at 0x1547afdd2a60>
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   communication_data_type ...... None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_pa
0: rameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}}
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   curriculum_enabled ........... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   curriculum_params ............ False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   dataloader_drop_last ......... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   disable_allgather ............ False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   dump_state ................... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   dynamic_loss_scale_args ...... None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_enabled ........... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_gas_boundary_resolution  1
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_layer_name ........ bert.encoder.layer
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_layer_num ......... 0
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_max_iter .......... 100
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_stability ......... 1e-06
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_tol ............... 0.01
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   eigenvalue_verbose ........... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   elasticity_enabled ........... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   flops_profiler_config ........ {
0:     "enabled": false, 
0:     "profile_step": 1, 
0:     "module_depth": -1, 
0:     "top_modules": 1, 
0:     "detailed": true, 
0:     "output_file": null
0: }
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   fp16_auto_cast ............... None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   fp16_enabled ................. False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   fp16_master_weights_and_gradients  False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   global_rank .................. 0
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   gradient_accumulation_steps .. 1
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   gradient_clipping ............ 1.0
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   gradient_predivide_factor .... 1.0
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   initial_dynamic_scale ........ 1
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   load_universal_checkpoint .... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   loss_scale ................... 1.0
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   memory_breakdown ............. False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   monitor_config ............... <deepspeed.monitor.config.DeepSpeedMonitorConfig object at 0x1547afdd29d0>
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   nebula_config ................ {
0:     "enabled": false, 
0:     "persistent_storage_path": null, 
0:     "persistent_time_interval": 100, 
0:     "num_of_version_in_retention": 2, 
0:     "enable_nebula_load": true, 
0:     "load_path": null
0: }
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   optimizer_legacy_fusion ...... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   optimizer_name ............... None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   optimizer_params ............. None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0}
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   pld_enabled .................. False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   pld_params ................... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   prescale_gradients ........... False
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   scheduler_name ............... None
0: [2023-03-16 23:24:12,597] [INFO] [config.py:1011:print]   scheduler_params ............. None
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   sparse_attention ............. None
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   sparse_gradients_enabled ..... False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   steps_per_print .............. 2000
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   train_batch_size ............. 256
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   train_micro_batch_size_per_gpu  4
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   use_node_local_storage ....... False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   wall_clock_breakdown ......... False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   world_size ................... 64
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   zero_allow_untested_optimizer  False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   zero_enabled ................. False
0: [2023-03-16 23:24:12,598] [INFO] [config.py:1011:print]   zero_optimization_stage ...... 0
0: [2023-03-16 23:24:12,598] [INFO] [config.py:996:print_user_config]   json = {
0:     "train_micro_batch_size_per_gpu": 4, 
0:     "train_batch_size": 256, 
0:     "gradient_clipping": 1.0, 
0:     "zero_optimization": {
0:         "stage": 0
0:     }, 
0:     "bf16": {
0:         "enabled": true
0:     }, 
0:     "steps_per_print": 2.000000e+03, 
0:     "wall_clock_breakdown": false
0: }
0: Time to load utils op: 0.0004260540008544922 seconds
0: [2023-03-16 23:24:12,598] [INFO] [engine.py:87:__init__] CONFIG: micro_batches=1 micro_batch_size=4
0: [2023-03-16 23:24:12,638] [INFO] [engine.py:145:__init__] RANK=0 STAGE=0 LAYERS=22 [0, 22) STAGE_PARAMS=146525952 (146.526M) TOTAL_PARAMS=146525952 (146.526M) UNIQUE_PARAMS=146525952 (146.526M)
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: WARNING: could not find the metadata file checkpoints_146m91b400m 
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0:     will not load any checkpoints and will start from random
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
0: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
4: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
6: [2023-03-16 23:24:12,645] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
3: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
2: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
5: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
1: [2023-03-16 23:24:12,646] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_146m91b400m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint.
7: time (ms) | load-checkpoint: 8.55
0: estimated model parameters: 0.146525952
0: estimated model parameters without embeddings: 0.106319616
0: [after model, optimizer, and learning rate scheduler are built] datetime: 2023-03-16 23:24:12 
0: > building train, validation, and test datasets ...
0:  > datasets target sizes (minimum size):
0:     train:      44416143
0:     validation: 4608
0:     test:       256
0: > building train, validation, and test datasets for GPT ...
0:  > building dataset index ...
0:     reading sizes...
0:     reading pointers...
0:     reading document index...
0:     creating numpy buffer of mmap...
0:     creating memory view of numpy buffer...
0:  > finished creating indexed dataset in 0.008498 seconds
0:     number of documents: 835726
0:  > dataset split:
0:     train:
0:      document indices in [0, 835726) total of 835726 documents
0:  > loading doc-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_44416143ns_2048sl_1234s_doc_idx.npy
0:  > loading sample-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_44416143ns_2048sl_1234s_sample_idx.npy
0:  > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_400M_text_document_train_indexmap_44416143ns_2048sl_1234s_shuffle_idx.npy
0:     loaded indexed file in 0.087 seconds
0:     total number of samples: 44482924
0:     total number of epochs: 228
0:  > building dataset index ...
0:     reading sizes...
0:     reading pointers...
0:     reading document index...
0:     creating numpy buffer of mmap...
0:     creating memory view of numpy buffer...
0:  > finished creating indexed dataset in 0.031906 seconds
0:     number of documents: 364608
0:  > dataset split:
0:     validation:
0:      document indices in [0, 364608) total of 364608 documents
0:  > loading doc-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_doc_idx.npy
0:  > loading sample-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_sample_idx.npy
0:  > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_4608ns_2048sl_1234s_shuffle_idx.npy
0:     loaded indexed file in 0.055 seconds
0:     total number of samples: 84978
0:     total number of epochs: 1
0: > finished creating GPT datasets ...
0: [after dataloaders are built] datetime: 2023-03-16 23:24:24 
0: done with setup ...
0: training ...
0: Number of parameters: [tensor rank - pipeline rank] w/ and w/o embeddings:
7: time (ms) | model-and-optimizer-setup: 16334.50 | train/valid/test-data-iterators-setup: 11104.38
0: [000-000] 0.1465B / 0.1063B
0: [before the start of training step] datetime: 2023-03-16 23:24:24 
0: [2023-03-16 23:24:25,110] [INFO] [checkpointing.py:553:forward] Activation Checkpointing Information
0: [2023-03-16 23:24:25,111] [INFO] [checkpointing.py:554:forward] ----Partition Activations False, CPU CHECKPOINTING False
0: [2023-03-16 23:24:25,111] [INFO] [checkpointing.py:557:forward] ----contiguous Memory Checkpointing False with None total layers
0: [2023-03-16 23:24:25,111] [INFO] [checkpointing.py:560:forward] ----Synchronization False
0: [2023-03-16 23:24:25,111] [INFO] [checkpointing.py:561:forward] ----Profiling time in checkpointing False
0: [Rank 0] (after 100 iterations) memory (MB) | allocated: 2731.17236328125 | max allocated: 5304.234375 | reserved: 6818.0 | max reserved: 6818.0
7:  iteration      100/  173500 | consumed samples:        25600 | consumed tokens:     52428800 | elapsed time per iteration (s): 0.49 | learning rate: 1.153E-05 | global batch size:   256 | lm loss: 9.645267E+00 | grad norm: 1.588 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 526.081 | TFLOPs: 24.56 |
7:  iteration      200/  173500 | consumed samples:        51200 | consumed tokens:    104857600 | elapsed time per iteration (s): 0.38 | learning rate: 2.305E-05 | global batch size:   256 | lm loss: 8.054749E+00 | grad norm: 0.998 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.692 | TFLOPs: 31.40 |
7:  iteration      300/  173500 | consumed samples:        76800 | consumed tokens:    157286400 | elapsed time per iteration (s): 0.38 | learning rate: 3.458E-05 | global batch size:   256 | lm loss: 7.063764E+00 | grad norm: 0.522 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.675 | TFLOPs: 31.72 |
7:  iteration      400/  173500 | consumed samples:       102400 | consumed tokens:    209715200 | elapsed time per iteration (s): 0.38 | learning rate: 4.611E-05 | global batch size:   256 | lm loss: 6.651832E+00 | grad norm: 0.517 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 675.116 | TFLOPs: 31.51 |
7:  iteration      500/  173500 | consumed samples:       128000 | consumed tokens:    262144000 | elapsed time per iteration (s): 0.38 | learning rate: 5.764E-05 | global batch size:   256 | lm loss: 6.421072E+00 | grad norm: 0.877 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.188 | TFLOPs: 31.42 |
7:  iteration      600/  173500 | consumed samples:       153600 | consumed tokens:    314572800 | elapsed time per iteration (s): 0.38 | learning rate: 6.916E-05 | global batch size:   256 | lm loss: 6.253424E+00 | grad norm: 1.267 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.136 | TFLOPs: 31.42 |
7:  iteration      700/  173500 | consumed samples:       179200 | consumed tokens:    367001600 | elapsed time per iteration (s): 0.38 | learning rate: 8.069E-05 | global batch size:   256 | lm loss: 6.095636E+00 | grad norm: 1.088 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 674.949 | TFLOPs: 31.50 |
7:  iteration      800/  173500 | consumed samples:       204800 | consumed tokens:    419430400 | elapsed time per iteration (s): 0.38 | learning rate: 9.222E-05 | global batch size:   256 | lm loss: 5.936118E+00 | grad norm: 1.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 671.193 | TFLOPs: 31.33 |
7:  iteration      900/  173500 | consumed samples:       230400 | consumed tokens:    471859200 | elapsed time per iteration (s): 0.38 | learning rate: 1.037E-04 | global batch size:   256 | lm loss: 5.786962E+00 | grad norm: 1.406 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 670.197 | TFLOPs: 31.28 |
7:  iteration     1000/  173500 | consumed samples:       256000 | consumed tokens:    524288000 | elapsed time per iteration (s): 0.37 | learning rate: 1.153E-04 | global batch size:   256 | lm loss: 5.633323E+00 | grad norm: 1.260 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.844 | TFLOPs: 31.97 |
7:  iteration     1100/  173500 | consumed samples:       281600 | consumed tokens:    576716800 | elapsed time per iteration (s): 0.37 | learning rate: 1.268E-04 | global batch size:   256 | lm loss: 5.485222E+00 | grad norm: 0.957 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.560 | TFLOPs: 32.14 |
7:  iteration     1200/  173500 | consumed samples:       307200 | consumed tokens:    629145600 | elapsed time per iteration (s): 0.37 | learning rate: 1.383E-04 | global batch size:   256 | lm loss: 5.338824E+00 | grad norm: 1.029 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.134 | TFLOPs: 32.07 |
7:  iteration     1300/  173500 | consumed samples:       332800 | consumed tokens:    681574400 | elapsed time per iteration (s): 0.37 | learning rate: 1.499E-04 | global batch size:   256 | lm loss: 5.192075E+00 | grad norm: 1.018 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.879 | TFLOPs: 32.01 |
7:  iteration     1400/  173500 | consumed samples:       358400 | consumed tokens:    734003200 | elapsed time per iteration (s): 0.37 | learning rate: 1.614E-04 | global batch size:   256 | lm loss: 5.029370E+00 | grad norm: 0.897 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.597 | TFLOPs: 32.14 |
7:  iteration     1500/  173500 | consumed samples:       384000 | consumed tokens:    786432000 | elapsed time per iteration (s): 0.37 | learning rate: 1.729E-04 | global batch size:   256 | lm loss: 4.870713E+00 | grad norm: 0.743 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.201 | TFLOPs: 32.22 |
7:  iteration     1600/  173500 | consumed samples:       409600 | consumed tokens:    838860800 | elapsed time per iteration (s): 0.37 | learning rate: 1.844E-04 | global batch size:   256 | lm loss: 4.749425E+00 | grad norm: 0.703 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.310 | TFLOPs: 32.31 |
7:  iteration     1700/  173500 | consumed samples:       435200 | consumed tokens:    891289600 | elapsed time per iteration (s): 0.37 | learning rate: 1.960E-04 | global batch size:   256 | lm loss: 4.656544E+00 | grad norm: 0.651 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.892 | TFLOPs: 32.25 |
7:  iteration     1800/  173500 | consumed samples:       460800 | consumed tokens:    943718400 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.578908E+00 | grad norm: 0.582 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.579 | TFLOPs: 32.19 |
7:  iteration     1900/  173500 | consumed samples:       486400 | consumed tokens:    996147200 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.507964E+00 | grad norm: 0.577 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.249 | TFLOPs: 32.17 |
0: [2023-03-16 23:37:06,466] [INFO] [logging.py:68:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.00019999894289482022, 0.00019999894289482022, 0.00019999894289482022], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     2000/  173500 | consumed samples:       512000 | consumed tokens:   1048576000 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.456920E+00 | grad norm: 0.701 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.167 | TFLOPs: 32.03 |
0: steps: 2000 loss: 4.4382 iter time (s): 0.379 samples/sec: 675.796
7:  iteration     2100/  173500 | consumed samples:       537600 | consumed tokens:   1101004800 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.411165E+00 | grad norm: 0.491 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.422 | TFLOPs: 31.90 |
7:  iteration     2200/  173500 | consumed samples:       563200 | consumed tokens:   1153433600 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.372287E+00 | grad norm: 0.761 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.039 | TFLOPs: 32.02 |
7:  iteration     2300/  173500 | consumed samples:       588800 | consumed tokens:   1205862400 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.332378E+00 | grad norm: 0.598 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.724 | TFLOPs: 32.19 |
7:  iteration     2400/  173500 | consumed samples:       614400 | consumed tokens:   1258291200 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.313095E+00 | grad norm: 0.463 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.538 | TFLOPs: 32.09 |
7:  iteration     2500/  173500 | consumed samples:       640000 | consumed tokens:   1310720000 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.270825E+00 | grad norm: 0.555 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.509 | TFLOPs: 32.09 |
7:  iteration     2600/  173500 | consumed samples:       665600 | consumed tokens:   1363148800 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.240795E+00 | grad norm: 0.381 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.458 | TFLOPs: 32.09 |
7:  iteration     2700/  173500 | consumed samples:       691200 | consumed tokens:   1415577600 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.216414E+00 | grad norm: 0.453 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.286 | TFLOPs: 32.08 |
7:  iteration     2800/  173500 | consumed samples:       716800 | consumed tokens:   1468006400 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.189811E+00 | grad norm: 0.509 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.987 | TFLOPs: 32.02 |
7:  iteration     2900/  173500 | consumed samples:       742400 | consumed tokens:   1520435200 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.168531E+00 | grad norm: 0.590 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.862 | TFLOPs: 31.92 |
7:  iteration     3000/  173500 | consumed samples:       768000 | consumed tokens:   1572864000 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.151326E+00 | grad norm: 0.441 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.209 | TFLOPs: 31.98 |
7:  iteration     3100/  173500 | consumed samples:       793600 | consumed tokens:   1625292800 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.128070E+00 | grad norm: 0.475 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.650 | TFLOPs: 32.28 |
7:  iteration     3200/  173500 | consumed samples:       819200 | consumed tokens:   1677721600 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.110559E+00 | grad norm: 0.466 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.069 | TFLOPs: 32.21 |
7:  iteration     3300/  173500 | consumed samples:       844800 | consumed tokens:   1730150400 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.091501E+00 | grad norm: 0.439 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.908 | TFLOPs: 32.11 |
7:  iteration     3400/  173500 | consumed samples:       870400 | consumed tokens:   1782579200 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.077614E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.927 | TFLOPs: 32.11 |
7:  iteration     3500/  173500 | consumed samples:       896000 | consumed tokens:   1835008000 | elapsed time per iteration (s): 0.37 | learning rate: 2.000E-04 | global batch size:   256 | lm loss: 4.060435E+00 | grad norm: 0.434 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.800 | TFLOPs: 32.29 |
7:  iteration     3600/  173500 | consumed samples:       921600 | consumed tokens:   1887436800 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 4.042073E+00 | grad norm: 0.461 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.457 | TFLOPs: 32.18 |
7:  iteration     3700/  173500 | consumed samples:       947200 | consumed tokens:   1939865600 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 4.029990E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.155 | TFLOPs: 32.35 |
7:  iteration     3800/  173500 | consumed samples:       972800 | consumed tokens:   1992294400 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 4.018772E+00 | grad norm: 0.401 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.123 | TFLOPs: 32.31 |
7:  iteration     3900/  173500 | consumed samples:       998400 | consumed tokens:   2044723200 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 4.004578E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.941 | TFLOPs: 32.39 |
0: [2023-03-16 23:49:29,802] [INFO] [logging.py:68:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.00019992278300259638, 0.00019992278300259638, 0.00019992278300259638], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     4000/  173500 | consumed samples:      1024000 | consumed tokens:   2097152000 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.987543E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.984 | TFLOPs: 32.39 |
0: steps: 4000 loss: 3.9902 iter time (s): 0.370 samples/sec: 692.561
7:  iteration     4100/  173500 | consumed samples:      1049600 | consumed tokens:   2149580800 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.980812E+00 | grad norm: 0.362 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.717 | TFLOPs: 32.43 |
7:  iteration     4200/  173500 | consumed samples:      1075200 | consumed tokens:   2202009600 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.968001E+00 | grad norm: 0.378 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.905 | TFLOPs: 32.39 |
7:  iteration     4300/  173500 | consumed samples:      1100800 | consumed tokens:   2254438400 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.955503E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.413 | TFLOPs: 32.46 |
7:  iteration     4400/  173500 | consumed samples:      1126400 | consumed tokens:   2306867200 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.943016E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.210 | TFLOPs: 32.31 |
7:  iteration     4500/  173500 | consumed samples:      1152000 | consumed tokens:   2359296000 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.934754E+00 | grad norm: 0.393 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.738 | TFLOPs: 32.43 |
7:  iteration     4600/  173500 | consumed samples:      1177600 | consumed tokens:   2411724800 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.924220E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.493 | TFLOPs: 32.37 |
7:  iteration     4700/  173500 | consumed samples:      1203200 | consumed tokens:   2464153600 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.912421E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.989 | TFLOPs: 32.35 |
7:  iteration     4800/  173500 | consumed samples:      1228800 | consumed tokens:   2516582400 | elapsed time per iteration (s): 0.37 | learning rate: 1.999E-04 | global batch size:   256 | lm loss: 3.905818E+00 | grad norm: 0.365 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.448 | TFLOPs: 32.37 |
7:  iteration     4900/  173500 | consumed samples:      1254400 | consumed tokens:   2569011200 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.901222E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.016 | TFLOPs: 32.44 |
7:  iteration     5000/  173500 | consumed samples:      1280000 | consumed tokens:   2621440000 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.888206E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.140 | TFLOPs: 32.40 |
7:  iteration     5100/  173500 | consumed samples:      1305600 | consumed tokens:   2673868800 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.878256E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.849 | TFLOPs: 32.43 |
7:  iteration     5200/  173500 | consumed samples:      1331200 | consumed tokens:   2726297600 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.872346E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.045 | TFLOPs: 32.44 |
7:  iteration     5300/  173500 | consumed samples:      1356800 | consumed tokens:   2778726400 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.864052E+00 | grad norm: 0.380 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.130 | TFLOPs: 32.45 |
7:  iteration     5400/  173500 | consumed samples:      1382400 | consumed tokens:   2831155200 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.857601E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.688 | TFLOPs: 32.47 |
7:  iteration     5500/  173500 | consumed samples:      1408000 | consumed tokens:   2883584000 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.851792E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.650 | TFLOPs: 32.47 |
7:  iteration     5600/  173500 | consumed samples:      1433600 | consumed tokens:   2936012800 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.843834E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.684 | TFLOPs: 32.43 |
7:  iteration     5700/  173500 | consumed samples:      1459200 | consumed tokens:   2988441600 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.835145E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.983 | TFLOPs: 32.44 |
7:  iteration     5800/  173500 | consumed samples:      1484800 | consumed tokens:   3040870400 | elapsed time per iteration (s): 0.37 | learning rate: 1.998E-04 | global batch size:   256 | lm loss: 3.833269E+00 | grad norm: 0.408 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.682 | TFLOPs: 32.43 |
7:  iteration     5900/  173500 | consumed samples:      1510400 | consumed tokens:   3093299200 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.827492E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.312 | TFLOPs: 32.36 |
0: [2023-03-17 00:01:47,065] [INFO] [logging.py:68:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0001997263111243839, 0.0001997263111243839, 0.0001997263111243839], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     6000/  173500 | consumed samples:      1536000 | consumed tokens:   3145728000 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.819527E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.136 | TFLOPs: 32.45 |
0: steps: 6000 loss: 3.7788 iter time (s): 0.367 samples/sec: 698.124
7:  iteration     6100/  173500 | consumed samples:      1561600 | consumed tokens:   3198156800 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.812390E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.163 | TFLOPs: 32.26 |
7:  iteration     6200/  173500 | consumed samples:      1587200 | consumed tokens:   3250585600 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.808438E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.130 | TFLOPs: 32.31 |
7:  iteration     6300/  173500 | consumed samples:      1612800 | consumed tokens:   3303014400 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.800560E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.048 | TFLOPs: 32.44 |
7:  iteration     6400/  173500 | consumed samples:      1638400 | consumed tokens:   3355443200 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.792065E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.114 | TFLOPs: 32.40 |
7:  iteration     6500/  173500 | consumed samples:      1664000 | consumed tokens:   3407872000 | elapsed time per iteration (s): 0.37 | learning rate: 1.997E-04 | global batch size:   256 | lm loss: 3.787817E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.214 | TFLOPs: 32.40 |
7:  iteration     6600/  173500 | consumed samples:      1689600 | consumed tokens:   3460300800 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.782657E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.943 | TFLOPs: 32.34 |
7:  iteration     6700/  173500 | consumed samples:      1715200 | consumed tokens:   3512729600 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.782091E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.907 | TFLOPs: 32.44 |
7:  iteration     6800/  173500 | consumed samples:      1740800 | consumed tokens:   3565158400 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.772401E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.410 | TFLOPs: 32.46 |
7:  iteration     6900/  173500 | consumed samples:      1766400 | consumed tokens:   3617587200 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.765802E+00 | grad norm: 0.275 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.014 | TFLOPs: 32.35 |
7:  iteration     7000/  173500 | consumed samples:      1792000 | consumed tokens:   3670016000 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.758839E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.961 | TFLOPs: 32.34 |
7:  iteration     7100/  173500 | consumed samples:      1817600 | consumed tokens:   3722444800 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.760417E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.650 | TFLOPs: 32.33 |
7:  iteration     7200/  173500 | consumed samples:      1843200 | consumed tokens:   3774873600 | elapsed time per iteration (s): 0.37 | learning rate: 1.996E-04 | global batch size:   256 | lm loss: 3.751738E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.884 | TFLOPs: 32.39 |
7:  iteration     7300/  173500 | consumed samples:      1868800 | consumed tokens:   3827302400 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 3.748513E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.953 | TFLOPs: 32.34 |
7:  iteration     7400/  173500 | consumed samples:      1894400 | consumed tokens:   3879731200 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 3.740302E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.086 | TFLOPs: 32.30 |
7:  iteration     7500/  173500 | consumed samples:      1920000 | consumed tokens:   3932160000 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 3.735158E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.242 | TFLOPs: 32.31 |
7:  iteration     7600/  173500 | consumed samples:      1945600 | consumed tokens:   3984588800 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 3.736122E+00 | grad norm: 0.259 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.508 | TFLOPs: 32.37 |
7:  iteration     7700/  173500 | consumed samples:      1971200 | consumed tokens:   4037017600 | elapsed time per iteration (s): 0.37 | learning rate: 1.995E-04 | global batch size:   256 | lm loss: 3.734704E+00 | grad norm: 0.268 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.214 | TFLOPs: 32.40 |
7:  iteration     7800/  173500 | consumed samples:      1996800 | consumed tokens:   4089446400 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.728292E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.158 | TFLOPs: 32.45 |
7:  iteration     7900/  173500 | consumed samples:      2022400 | consumed tokens:   4141875200 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.727584E+00 | grad norm: 0.266 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.118 | TFLOPs: 32.45 |
0: [2023-03-17 00:14:05,209] [INFO] [logging.py:68:log_dist] [Rank 0] step=8000, skipped=0, lr=[0.00019940979012929202, 0.00019940979012929202, 0.00019940979012929202], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration     8000/  173500 | consumed samples:      2048000 | consumed tokens:   4194304000 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.721721E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.971 | TFLOPs: 32.44 |
0: steps: 8000 loss: 3.7511 iter time (s): 0.367 samples/sec: 697.310
7:  iteration     8100/  173500 | consumed samples:      2073600 | consumed tokens:   4246732800 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.714159E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.828 | TFLOPs: 32.43 |
7:  iteration     8200/  173500 | consumed samples:      2099200 | consumed tokens:   4299161600 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.706900E+00 | grad norm: 0.282 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.055 | TFLOPs: 32.44 |
7:  iteration     8300/  173500 | consumed samples:      2124800 | consumed tokens:   4351590400 | elapsed time per iteration (s): 0.37 | learning rate: 1.994E-04 | global batch size:   256 | lm loss: 3.706071E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.059 | TFLOPs: 32.44 |
7:  iteration     8400/  173500 | consumed samples:      2150400 | consumed tokens:   4404019200 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 3.703886E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.781 | TFLOPs: 32.43 |
7:  iteration     8500/  173500 | consumed samples:      2176000 | consumed tokens:   4456448000 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 3.697773E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.988 | TFLOPs: 32.44 |
7:  iteration     8600/  173500 | consumed samples:      2201600 | consumed tokens:   4508876800 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 3.694155E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.066 | TFLOPs: 32.44 |
7:  iteration     8700/  173500 | consumed samples:      2227200 | consumed tokens:   4561305600 | elapsed time per iteration (s): 0.37 | learning rate: 1.993E-04 | global batch size:   256 | lm loss: 3.691778E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.274 | TFLOPs: 32.45 |
7:  iteration     8800/  173500 | consumed samples:      2252800 | consumed tokens:   4613734400 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 3.687705E+00 | grad norm: 0.252 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.379 | TFLOPs: 32.46 |
7:  iteration     8900/  173500 | consumed samples:      2278400 | consumed tokens:   4666163200 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 3.687765E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.228 | TFLOPs: 32.45 |
7:  iteration     9000/  173500 | consumed samples:      2304000 | consumed tokens:   4718592000 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 3.675754E+00 | grad norm: 0.246 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.452 | TFLOPs: 32.46 |
7:  iteration     9100/  173500 | consumed samples:      2329600 | consumed tokens:   4771020800 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 3.677835E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.418 | TFLOPs: 32.46 |
7:  iteration     9200/  173500 | consumed samples:      2355200 | consumed tokens:   4823449600 | elapsed time per iteration (s): 0.37 | learning rate: 1.992E-04 | global batch size:   256 | lm loss: 3.670468E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.200 | TFLOPs: 32.45 |
7:  iteration     9300/  173500 | consumed samples:      2380800 | consumed tokens:   4875878400 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 3.669695E+00 | grad norm: 0.273 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.190 | TFLOPs: 32.45 |
7:  iteration     9400/  173500 | consumed samples:      2406400 | consumed tokens:   4928307200 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 3.664136E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.098 | TFLOPs: 32.40 |
7:  iteration     9500/  173500 | consumed samples:      2432000 | consumed tokens:   4980736000 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 3.661853E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.833 | TFLOPs: 32.43 |
7:  iteration     9600/  173500 | consumed samples:      2457600 | consumed tokens:   5033164800 | elapsed time per iteration (s): 0.37 | learning rate: 1.991E-04 | global batch size:   256 | lm loss: 3.662213E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.820 | TFLOPs: 32.43 |
7:  iteration     9700/  173500 | consumed samples:      2483200 | consumed tokens:   5085593600 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 3.687735E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.150 | TFLOPs: 32.31 |
7:  iteration     9800/  173500 | consumed samples:      2508800 | consumed tokens:   5138022400 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 3.660412E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.779 | TFLOPs: 32.38 |
7:  iteration     9900/  173500 | consumed samples:      2534400 | consumed tokens:   5190451200 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 3.653995E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.617 | TFLOPs: 32.42 |
0: [2023-03-17 00:26:22,130] [INFO] [logging.py:68:log_dist] [Rank 0] step=10000, skipped=0, lr=[0.00019897364350587667, 0.00019897364350587667, 0.00019897364350587667], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    10000/  173500 | consumed samples:      2560000 | consumed tokens:   5242880000 | elapsed time per iteration (s): 0.37 | learning rate: 1.990E-04 | global batch size:   256 | lm loss: 3.647867E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.452 | TFLOPs: 32.41 |
0: steps: 10000 loss: 3.6233 iter time (s): 0.366 samples/sec: 698.508
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 10000 | lm loss value: 3.679149E+00 | lm loss PPL: 3.961267E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   10000 to checkpoints_146m91b400m
0: [2023-03-17 00:26:22,253] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step10000 is begin to save!
0: [2023-03-17 00:26:22,272] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_01-model_00-model_states.pt...
0: [2023-03-17 00:26:22,361] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_01-model_00-model_states.pt.
0: [2023-03-17 00:26:22,362] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_03-model_00-model_states.pt...
0: [2023-03-17 00:26:22,378] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_03-model_00-model_states.pt.
0: [2023-03-17 00:26:22,378] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_04-model_00-model_states.pt...
0: [2023-03-17 00:26:22,394] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_04-model_00-model_states.pt.
0: [2023-03-17 00:26:22,395] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_05-model_00-model_states.pt...
0: [2023-03-17 00:26:22,410] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_05-model_00-model_states.pt.
0: [2023-03-17 00:26:22,410] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_06-model_00-model_states.pt...
0: [2023-03-17 00:26:22,425] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_06-model_00-model_states.pt.
0: [2023-03-17 00:26:22,425] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_07-model_00-model_states.pt...
0: [2023-03-17 00:26:22,440] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_07-model_00-model_states.pt.
0: [2023-03-17 00:26:22,441] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_08-model_00-model_states.pt...
0: [2023-03-17 00:26:22,456] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_08-model_00-model_states.pt.
0: [2023-03-17 00:26:22,456] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_09-model_00-model_states.pt...
0: [2023-03-17 00:26:22,471] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_09-model_00-model_states.pt.
0: [2023-03-17 00:26:22,471] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_10-model_00-model_states.pt...
0: [2023-03-17 00:26:22,486] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_10-model_00-model_states.pt.
0: [2023-03-17 00:26:22,486] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_11-model_00-model_states.pt...
0: [2023-03-17 00:26:22,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_11-model_00-model_states.pt.
0: [2023-03-17 00:26:22,501] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_12-model_00-model_states.pt...
0: [2023-03-17 00:26:22,516] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_12-model_00-model_states.pt.
0: [2023-03-17 00:26:22,516] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_13-model_00-model_states.pt...
0: [2023-03-17 00:26:22,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_13-model_00-model_states.pt.
0: [2023-03-17 00:26:22,532] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_14-model_00-model_states.pt...
0: [2023-03-17 00:26:22,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_14-model_00-model_states.pt.
0: [2023-03-17 00:26:22,547] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_15-model_00-model_states.pt...
0: [2023-03-17 00:26:22,562] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_15-model_00-model_states.pt.
0: [2023-03-17 00:26:22,562] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_16-model_00-model_states.pt...
0: [2023-03-17 00:26:22,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_16-model_00-model_states.pt.
0: [2023-03-17 00:26:22,577] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_17-model_00-model_states.pt...
0: [2023-03-17 00:26:22,592] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_17-model_00-model_states.pt.
0: [2023-03-17 00:26:22,592] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/layer_19-model_00-model_states.pt...
0: [2023-03-17 00:26:22,593] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/layer_19-model_00-model_states.pt.
0: [2023-03-17 00:26:22,594] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step10000/mp_rank_00_model_states.pt
0: [2023-03-17 00:26:22,594] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/mp_rank_00_model_states.pt...
0: [2023-03-17 00:26:22,597] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/mp_rank_00_model_states.pt.
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
7: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
2: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 00:26:22,616] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
0: [2023-03-17 00:26:22,653] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,654] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,654] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,654] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,654] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,655] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,655] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,656] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 00:26:22,656] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
2: [2023-03-17 00:26:22,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 00:26:22,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,664] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,666] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,666] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,666] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,667] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,667] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,667] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,667] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,667] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
3: [2023-03-17 00:26:22,667] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 00:26:22,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 00:26:22,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,668] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,668] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,668] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,669] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,669] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
6: [2023-03-17 00:26:22,670] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 00:26:22,670] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 00:26:22,670] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,673] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,673] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,674] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,674] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,675] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,675] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,675] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
4: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
7: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
4: [2023-03-17 00:26:22,676] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
7: [2023-03-17 00:26:22,676] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
4: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
7: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
1: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 00:26:22,676] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 00:26:22,676] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 00:26:22,680] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,680] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,680] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
5: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0: [2023-03-17 00:26:22,680] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 00:26:22,680] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now!
0:   successfully saved checkpoint at iteration   10000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 432.75
7:  iteration    10100/  173500 | consumed samples:      2585600 | consumed tokens:   5295308800 | elapsed time per iteration (s): 0.38 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 3.642037E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.896 | TFLOPs: 31.78 |
7:  iteration    10200/  173500 | consumed samples:      2611200 | consumed tokens:   5347737600 | elapsed time per iteration (s): 0.37 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 3.641638E+00 | grad norm: 0.252 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.378 | TFLOPs: 32.41 |
7:  iteration    10300/  173500 | consumed samples:      2636800 | consumed tokens:   5400166400 | elapsed time per iteration (s): 0.38 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 3.638981E+00 | grad norm: 0.282 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.257 | TFLOPs: 31.85 |
7:  iteration    10400/  173500 | consumed samples:      2662400 | consumed tokens:   5452595200 | elapsed time per iteration (s): 0.37 | learning rate: 1.989E-04 | global batch size:   256 | lm loss: 3.638584E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.959 | TFLOPs: 31.92 |
7:  iteration    10500/  173500 | consumed samples:      2688000 | consumed tokens:   5505024000 | elapsed time per iteration (s): 0.38 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 3.633100E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 666.607 | TFLOPs: 31.11 |
7:  iteration    10600/  173500 | consumed samples:      2713600 | consumed tokens:   5557452800 | elapsed time per iteration (s): 0.37 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 3.630028E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.140 | TFLOPs: 31.98 |
7:  iteration    10700/  173500 | consumed samples:      2739200 | consumed tokens:   5609881600 | elapsed time per iteration (s): 0.37 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 3.623980E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.087 | TFLOPs: 31.98 |
7:  iteration    10800/  173500 | consumed samples:      2764800 | consumed tokens:   5662310400 | elapsed time per iteration (s): 0.37 | learning rate: 1.988E-04 | global batch size:   256 | lm loss: 3.624683E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.866 | TFLOPs: 31.97 |
7:  iteration    10900/  173500 | consumed samples:      2790400 | consumed tokens:   5714739200 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 3.625418E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.716 | TFLOPs: 31.96 |
7:  iteration    11000/  173500 | consumed samples:      2816000 | consumed tokens:   5767168000 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 3.613610E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.261 | TFLOPs: 32.22 |
7:  iteration    11100/  173500 | consumed samples:      2841600 | consumed tokens:   5819596800 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 3.620206E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.557 | TFLOPs: 32.05 |
7:  iteration    11200/  173500 | consumed samples:      2867200 | consumed tokens:   5872025600 | elapsed time per iteration (s): 0.37 | learning rate: 1.987E-04 | global batch size:   256 | lm loss: 3.618164E+00 | grad norm: 0.266 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.599 | TFLOPs: 32.00 |
7:  iteration    11300/  173500 | consumed samples:      2892800 | consumed tokens:   5924454400 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 3.612151E+00 | grad norm: 0.255 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.774 | TFLOPs: 32.20 |
7:  iteration    11400/  173500 | consumed samples:      2918400 | consumed tokens:   5976883200 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 3.613426E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.729 | TFLOPs: 32.05 |
7:  iteration    11500/  173500 | consumed samples:      2944000 | consumed tokens:   6029312000 | elapsed time per iteration (s): 0.37 | learning rate: 1.986E-04 | global batch size:   256 | lm loss: 3.607449E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.033 | TFLOPs: 32.25 |
7:  iteration    11600/  173500 | consumed samples:      2969600 | consumed tokens:   6081740800 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size:   256 | lm loss: 3.606198E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.860 | TFLOPs: 32.29 |
7:  iteration    11700/  173500 | consumed samples:      2995200 | consumed tokens:   6134169600 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size:   256 | lm loss: 3.607420E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.531 | TFLOPs: 32.37 |
7:  iteration    11800/  173500 | consumed samples:      3020800 | consumed tokens:   6186598400 | elapsed time per iteration (s): 0.37 | learning rate: 1.985E-04 | global batch size:   256 | lm loss: 3.601948E+00 | grad norm: 0.266 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.935 | TFLOPs: 32.39 |
7:  iteration    11900/  173500 | consumed samples:      3046400 | consumed tokens:   6239027200 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 3.601476E+00 | grad norm: 0.247 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.435 | TFLOPs: 32.37 |
0: [2023-03-17 00:38:47,282] [INFO] [logging.py:68:log_dist] [Rank 0] step=12000, skipped=0, lr=[0.0001984184547955352, 0.0001984184547955352, 0.0001984184547955352], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    12000/  173500 | consumed samples:      3072000 | consumed tokens:   6291456000 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 3.598621E+00 | grad norm: 0.282 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.721 | TFLOPs: 32.33 |
0: steps: 12000 loss: 3.5743 iter time (s): 0.370 samples/sec: 691.036
7:  iteration    12100/  173500 | consumed samples:      3097600 | consumed tokens:   6343884800 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 3.590966E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.770 | TFLOPs: 32.38 |
7:  iteration    12200/  173500 | consumed samples:      3123200 | consumed tokens:   6396313600 | elapsed time per iteration (s): 0.37 | learning rate: 1.984E-04 | global batch size:   256 | lm loss: 3.590447E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.063 | TFLOPs: 32.30 |
7:  iteration    12300/  173500 | consumed samples:      3148800 | consumed tokens:   6448742400 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size:   256 | lm loss: 3.588095E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.794 | TFLOPs: 32.38 |
7:  iteration    12400/  173500 | consumed samples:      3174400 | consumed tokens:   6501171200 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size:   256 | lm loss: 3.589153E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.679 | TFLOPs: 32.38 |
7:  iteration    12500/  173500 | consumed samples:      3200000 | consumed tokens:   6553600000 | elapsed time per iteration (s): 0.37 | learning rate: 1.983E-04 | global batch size:   256 | lm loss: 3.587365E+00 | grad norm: 0.275 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.046 | TFLOPs: 32.40 |
7:  iteration    12600/  173500 | consumed samples:      3225600 | consumed tokens:   6606028800 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size:   256 | lm loss: 3.582046E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.096 | TFLOPs: 32.40 |
7:  iteration    12700/  173500 | consumed samples:      3251200 | consumed tokens:   6658457600 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size:   256 | lm loss: 3.584843E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.248 | TFLOPs: 32.40 |
7:  iteration    12800/  173500 | consumed samples:      3276800 | consumed tokens:   6710886400 | elapsed time per iteration (s): 0.37 | learning rate: 1.982E-04 | global batch size:   256 | lm loss: 3.580710E+00 | grad norm: 0.250 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.825 | TFLOPs: 32.39 |
7:  iteration    12900/  173500 | consumed samples:      3302400 | consumed tokens:   6763315200 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size:   256 | lm loss: 3.576869E+00 | grad norm: 0.272 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.831 | TFLOPs: 32.34 |
7:  iteration    13000/  173500 | consumed samples:      3328000 | consumed tokens:   6815744000 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size:   256 | lm loss: 3.579746E+00 | grad norm: 0.261 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.133 | TFLOPs: 32.40 |
7:  iteration    13100/  173500 | consumed samples:      3353600 | consumed tokens:   6868172800 | elapsed time per iteration (s): 0.37 | learning rate: 1.981E-04 | global batch size:   256 | lm loss: 3.570008E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.606 | TFLOPs: 32.42 |
7:  iteration    13200/  173500 | consumed samples:      3379200 | consumed tokens:   6920601600 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size:   256 | lm loss: 3.572318E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.648 | TFLOPs: 32.42 |
7:  iteration    13300/  173500 | consumed samples:      3404800 | consumed tokens:   6973030400 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size:   256 | lm loss: 3.570410E+00 | grad norm: 0.248 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.695 | TFLOPs: 32.43 |
7:  iteration    13400/  173500 | consumed samples:      3430400 | consumed tokens:   7025459200 | elapsed time per iteration (s): 0.37 | learning rate: 1.980E-04 | global batch size:   256 | lm loss: 3.568824E+00 | grad norm: 0.258 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.995 | TFLOPs: 32.39 |
7:  iteration    13500/  173500 | consumed samples:      3456000 | consumed tokens:   7077888000 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size:   256 | lm loss: 3.566461E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.625 | TFLOPs: 32.33 |
7:  iteration    13600/  173500 | consumed samples:      3481600 | consumed tokens:   7130316800 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size:   256 | lm loss: 3.563235E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.477 | TFLOPs: 32.37 |
7:  iteration    13700/  173500 | consumed samples:      3507200 | consumed tokens:   7182745600 | elapsed time per iteration (s): 0.37 | learning rate: 1.979E-04 | global batch size:   256 | lm loss: 3.561401E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.334 | TFLOPs: 32.36 |
7:  iteration    13800/  173500 | consumed samples:      3532800 | consumed tokens:   7235174400 | elapsed time per iteration (s): 0.37 | learning rate: 1.978E-04 | global batch size:   256 | lm loss: 3.562463E+00 | grad norm: 0.238 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.324 | TFLOPs: 32.36 |
7:  iteration    13900/  173500 | consumed samples:      3558400 | consumed tokens:   7287603200 | elapsed time per iteration (s): 0.37 | learning rate: 1.978E-04 | global batch size:   256 | lm loss: 3.563369E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.963 | TFLOPs: 32.34 |
0: [2023-03-17 00:51:05,378] [INFO] [logging.py:68:log_dist] [Rank 0] step=14000, skipped=0, lr=[0.00019774496681175836, 0.00019774496681175836, 0.00019774496681175836], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    14000/  173500 | consumed samples:      3584000 | consumed tokens:   7340032000 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size:   256 | lm loss: 3.555647E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.383 | TFLOPs: 32.36 |
0: steps: 14000 loss: 3.5509 iter time (s): 0.367 samples/sec: 697.254
7:  iteration    14100/  173500 | consumed samples:      3609600 | consumed tokens:   7392460800 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size:   256 | lm loss: 3.554185E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.163 | TFLOPs: 32.35 |
7:  iteration    14200/  173500 | consumed samples:      3635200 | consumed tokens:   7444889600 | elapsed time per iteration (s): 0.37 | learning rate: 1.977E-04 | global batch size:   256 | lm loss: 3.556831E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.105 | TFLOPs: 32.40 |
7:  iteration    14300/  173500 | consumed samples:      3660800 | consumed tokens:   7497318400 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size:   256 | lm loss: 3.550753E+00 | grad norm: 0.246 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.373 | TFLOPs: 32.36 |
7:  iteration    14400/  173500 | consumed samples:      3686400 | consumed tokens:   7549747200 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size:   256 | lm loss: 3.552695E+00 | grad norm: 0.262 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.787 | TFLOPs: 32.38 |
7:  iteration    14500/  173500 | consumed samples:      3712000 | consumed tokens:   7602176000 | elapsed time per iteration (s): 0.37 | learning rate: 1.976E-04 | global batch size:   256 | lm loss: 3.547706E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.912 | TFLOPs: 32.39 |
7:  iteration    14600/  173500 | consumed samples:      3737600 | consumed tokens:   7654604800 | elapsed time per iteration (s): 0.37 | learning rate: 1.975E-04 | global batch size:   256 | lm loss: 3.545429E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.572 | TFLOPs: 32.37 |
7:  iteration    14700/  173500 | consumed samples:      3763200 | consumed tokens:   7707033600 | elapsed time per iteration (s): 0.37 | learning rate: 1.975E-04 | global batch size:   256 | lm loss: 3.544378E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.097 | TFLOPs: 32.40 |
7:  iteration    14800/  173500 | consumed samples:      3788800 | consumed tokens:   7759462400 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size:   256 | lm loss: 3.543870E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.201 | TFLOPs: 32.36 |
7:  iteration    14900/  173500 | consumed samples:      3814400 | consumed tokens:   7811891200 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size:   256 | lm loss: 3.543255E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.473 | TFLOPs: 32.37 |
7:  iteration    15000/  173500 | consumed samples:      3840000 | consumed tokens:   7864320000 | elapsed time per iteration (s): 0.37 | learning rate: 1.974E-04 | global batch size:   256 | lm loss: 3.538685E+00 | grad norm: 0.265 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.736 | TFLOPs: 32.38 |
7:  iteration    15100/  173500 | consumed samples:      3865600 | consumed tokens:   7916748800 | elapsed time per iteration (s): 0.37 | learning rate: 1.973E-04 | global batch size:   256 | lm loss: 3.540468E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.786 | TFLOPs: 32.38 |
7:  iteration    15200/  173500 | consumed samples:      3891200 | consumed tokens:   7969177600 | elapsed time per iteration (s): 0.37 | learning rate: 1.973E-04 | global batch size:   256 | lm loss: 3.542986E+00 | grad norm: 0.251 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.841 | TFLOPs: 32.39 |
7:  iteration    15300/  173500 | consumed samples:      3916800 | consumed tokens:   8021606400 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size:   256 | lm loss: 3.535972E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.776 | TFLOPs: 32.34 |
7:  iteration    15400/  173500 | consumed samples:      3942400 | consumed tokens:   8074035200 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size:   256 | lm loss: 3.533015E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.371 | TFLOPs: 32.36 |
7:  iteration    15500/  173500 | consumed samples:      3968000 | consumed tokens:   8126464000 | elapsed time per iteration (s): 0.37 | learning rate: 1.972E-04 | global batch size:   256 | lm loss: 3.534358E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.767 | TFLOPs: 32.38 |
7:  iteration    15600/  173500 | consumed samples:      3993600 | consumed tokens:   8178892800 | elapsed time per iteration (s): 0.37 | learning rate: 1.971E-04 | global batch size:   256 | lm loss: 3.534134E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.805 | TFLOPs: 32.38 |
7:  iteration    15700/  173500 | consumed samples:      4019200 | consumed tokens:   8231321600 | elapsed time per iteration (s): 0.37 | learning rate: 1.971E-04 | global batch size:   256 | lm loss: 3.523377E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.555 | TFLOPs: 32.37 |
7:  iteration    15800/  173500 | consumed samples:      4044800 | consumed tokens:   8283750400 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size:   256 | lm loss: 3.529844E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.672 | TFLOPs: 32.38 |
7:  iteration    15900/  173500 | consumed samples:      4070400 | consumed tokens:   8336179200 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size:   256 | lm loss: 3.532377E+00 | grad norm: 0.253 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.976 | TFLOPs: 32.39 |
0: [2023-03-17 01:03:23,479] [INFO] [logging.py:68:log_dist] [Rank 0] step=16000, skipped=0, lr=[0.00019695408064628468, 0.00019695408064628468, 0.00019695408064628468], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    16000/  173500 | consumed samples:      4096000 | consumed tokens:   8388608000 | elapsed time per iteration (s): 0.37 | learning rate: 1.970E-04 | global batch size:   256 | lm loss: 3.523707E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.482 | TFLOPs: 32.42 |
0: steps: 16000 loss: 3.5506 iter time (s): 0.367 samples/sec: 697.421
7:  iteration    16100/  173500 | consumed samples:      4121600 | consumed tokens:   8441036800 | elapsed time per iteration (s): 0.37 | learning rate: 1.969E-04 | global batch size:   256 | lm loss: 3.529183E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.670 | TFLOPs: 32.38 |
7:  iteration    16200/  173500 | consumed samples:      4147200 | consumed tokens:   8493465600 | elapsed time per iteration (s): 0.37 | learning rate: 1.969E-04 | global batch size:   256 | lm loss: 3.527337E+00 | grad norm: 0.246 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.579 | TFLOPs: 32.37 |
7:  iteration    16300/  173500 | consumed samples:      4172800 | consumed tokens:   8545894400 | elapsed time per iteration (s): 0.37 | learning rate: 1.968E-04 | global batch size:   256 | lm loss: 3.521430E+00 | grad norm: 0.248 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.508 | TFLOPs: 32.37 |
7:  iteration    16400/  173500 | consumed samples:      4198400 | consumed tokens:   8598323200 | elapsed time per iteration (s): 0.37 | learning rate: 1.968E-04 | global batch size:   256 | lm loss: 3.516093E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.953 | TFLOPs: 32.34 |
7:  iteration    16500/  173500 | consumed samples:      4224000 | consumed tokens:   8650752000 | elapsed time per iteration (s): 0.37 | learning rate: 1.967E-04 | global batch size:   256 | lm loss: 3.522067E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.395 | TFLOPs: 32.32 |
7:  iteration    16600/  173500 | consumed samples:      4249600 | consumed tokens:   8703180800 | elapsed time per iteration (s): 0.37 | learning rate: 1.967E-04 | global batch size:   256 | lm loss: 3.522863E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.427 | TFLOPs: 32.37 |
7:  iteration    16700/  173500 | consumed samples:      4275200 | consumed tokens:   8755609600 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size:   256 | lm loss: 3.515405E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.107 | TFLOPs: 32.35 |
7:  iteration    16800/  173500 | consumed samples:      4300800 | consumed tokens:   8808038400 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size:   256 | lm loss: 3.521105E+00 | grad norm: 0.251 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.824 | TFLOPs: 32.34 |
7:  iteration    16900/  173500 | consumed samples:      4326400 | consumed tokens:   8860467200 | elapsed time per iteration (s): 0.37 | learning rate: 1.966E-04 | global batch size:   256 | lm loss: 3.517272E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.831 | TFLOPs: 32.34 |
7:  iteration    17000/  173500 | consumed samples:      4352000 | consumed tokens:   8912896000 | elapsed time per iteration (s): 0.37 | learning rate: 1.965E-04 | global batch size:   256 | lm loss: 3.513228E+00 | grad norm: 0.252 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.621 | TFLOPs: 32.33 |
7:  iteration    17100/  173500 | consumed samples:      4377600 | consumed tokens:   8965324800 | elapsed time per iteration (s): 0.37 | learning rate: 1.965E-04 | global batch size:   256 | lm loss: 3.515558E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.899 | TFLOPs: 32.30 |
7:  iteration    17200/  173500 | consumed samples:      4403200 | consumed tokens:   9017753600 | elapsed time per iteration (s): 0.37 | learning rate: 1.964E-04 | global batch size:   256 | lm loss: 3.511805E+00 | grad norm: 0.253 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.435 | TFLOPs: 32.37 |
7:  iteration    17300/  173500 | consumed samples:      4428800 | consumed tokens:   9070182400 | elapsed time per iteration (s): 0.37 | learning rate: 1.964E-04 | global batch size:   256 | lm loss: 3.512533E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.672 | TFLOPs: 32.38 |
7:  iteration    17400/  173500 | consumed samples:      4454400 | consumed tokens:   9122611200 | elapsed time per iteration (s): 0.37 | learning rate: 1.963E-04 | global batch size:   256 | lm loss: 3.508640E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.727 | TFLOPs: 32.38 |
7:  iteration    17500/  173500 | consumed samples:      4480000 | consumed tokens:   9175040000 | elapsed time per iteration (s): 0.37 | learning rate: 1.963E-04 | global batch size:   256 | lm loss: 3.503698E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.720 | TFLOPs: 32.38 |
7:  iteration    17600/  173500 | consumed samples:      4505600 | consumed tokens:   9227468800 | elapsed time per iteration (s): 0.37 | learning rate: 1.962E-04 | global batch size:   256 | lm loss: 3.504434E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.628 | TFLOPs: 32.38 |
7:  iteration    17700/  173500 | consumed samples:      4531200 | consumed tokens:   9279897600 | elapsed time per iteration (s): 0.37 | learning rate: 1.962E-04 | global batch size:   256 | lm loss: 3.503867E+00 | grad norm: 0.264 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.607 | TFLOPs: 32.38 |
7:  iteration    17800/  173500 | consumed samples:      4556800 | consumed tokens:   9332326400 | elapsed time per iteration (s): 0.37 | learning rate: 1.961E-04 | global batch size:   256 | lm loss: 3.505687E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.293 | TFLOPs: 32.36 |
7:  iteration    17900/  173500 | consumed samples:      4582400 | consumed tokens:   9384755200 | elapsed time per iteration (s): 0.37 | learning rate: 1.961E-04 | global batch size:   256 | lm loss: 3.504364E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.015 | TFLOPs: 32.35 |
0: [2023-03-17 01:15:42,038] [INFO] [logging.py:68:log_dist] [Rank 0] step=18000, skipped=0, lr=[0.00019604685446348677, 0.00019604685446348677, 0.00019604685446348677], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    18000/  173500 | consumed samples:      4608000 | consumed tokens:   9437184000 | elapsed time per iteration (s): 0.37 | learning rate: 1.960E-04 | global batch size:   256 | lm loss: 3.502303E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.959 | TFLOPs: 32.39 |
0: steps: 18000 loss: 3.4825 iter time (s): 0.367 samples/sec: 697.089
7:  iteration    18100/  173500 | consumed samples:      4633600 | consumed tokens:   9489612800 | elapsed time per iteration (s): 0.37 | learning rate: 1.960E-04 | global batch size:   256 | lm loss: 3.501749E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.200 | TFLOPs: 32.36 |
7:  iteration    18200/  173500 | consumed samples:      4659200 | consumed tokens:   9542041600 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size:   256 | lm loss: 3.496257E+00 | grad norm: 0.238 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.001 | TFLOPs: 32.39 |
7:  iteration    18300/  173500 | consumed samples:      4684800 | consumed tokens:   9594470400 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size:   256 | lm loss: 3.500969E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.042 | TFLOPs: 32.40 |
7:  iteration    18400/  173500 | consumed samples:      4710400 | consumed tokens:   9646899200 | elapsed time per iteration (s): 0.37 | learning rate: 1.959E-04 | global batch size:   256 | lm loss: 3.496497E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.674 | TFLOPs: 32.38 |
7:  iteration    18500/  173500 | consumed samples:      4736000 | consumed tokens:   9699328000 | elapsed time per iteration (s): 0.37 | learning rate: 1.958E-04 | global batch size:   256 | lm loss: 3.496192E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.800 | TFLOPs: 32.38 |
7:  iteration    18600/  173500 | consumed samples:      4761600 | consumed tokens:   9751756800 | elapsed time per iteration (s): 0.37 | learning rate: 1.958E-04 | global batch size:   256 | lm loss: 3.494166E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.820 | TFLOPs: 32.38 |
7:  iteration    18700/  173500 | consumed samples:      4787200 | consumed tokens:   9804185600 | elapsed time per iteration (s): 0.37 | learning rate: 1.957E-04 | global batch size:   256 | lm loss: 3.493619E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.044 | TFLOPs: 32.40 |
7:  iteration    18800/  173500 | consumed samples:      4812800 | consumed tokens:   9856614400 | elapsed time per iteration (s): 0.37 | learning rate: 1.957E-04 | global batch size:   256 | lm loss: 3.490758E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.526 | TFLOPs: 32.37 |
7:  iteration    18900/  173500 | consumed samples:      4838400 | consumed tokens:   9909043200 | elapsed time per iteration (s): 0.37 | learning rate: 1.956E-04 | global batch size:   256 | lm loss: 3.492078E+00 | grad norm: 0.247 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.085 | TFLOPs: 32.35 |
7:  iteration    19000/  173500 | consumed samples:      4864000 | consumed tokens:   9961472000 | elapsed time per iteration (s): 0.37 | learning rate: 1.955E-04 | global batch size:   256 | lm loss: 3.490706E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.545 | TFLOPs: 32.14 |
7:  iteration    19100/  173500 | consumed samples:      4889600 | consumed tokens:  10013900800 | elapsed time per iteration (s): 0.37 | learning rate: 1.955E-04 | global batch size:   256 | lm loss: 3.487851E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.224 | TFLOPs: 32.36 |
7:  iteration    19200/  173500 | consumed samples:      4915200 | consumed tokens:  10066329600 | elapsed time per iteration (s): 0.37 | learning rate: 1.954E-04 | global batch size:   256 | lm loss: 3.488707E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.483 | TFLOPs: 32.37 |
7:  iteration    19300/  173500 | consumed samples:      4940800 | consumed tokens:  10118758400 | elapsed time per iteration (s): 0.37 | learning rate: 1.954E-04 | global batch size:   256 | lm loss: 3.488395E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.762 | TFLOPs: 32.38 |
7:  iteration    19400/  173500 | consumed samples:      4966400 | consumed tokens:  10171187200 | elapsed time per iteration (s): 0.37 | learning rate: 1.953E-04 | global batch size:   256 | lm loss: 3.490943E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.844 | TFLOPs: 32.39 |
7:  iteration    19500/  173500 | consumed samples:      4992000 | consumed tokens:  10223616000 | elapsed time per iteration (s): 0.37 | learning rate: 1.953E-04 | global batch size:   256 | lm loss: 3.489446E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.883 | TFLOPs: 32.39 |
7:  iteration    19600/  173500 | consumed samples:      5017600 | consumed tokens:  10276044800 | elapsed time per iteration (s): 0.37 | learning rate: 1.952E-04 | global batch size:   256 | lm loss: 3.488992E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.210 | TFLOPs: 31.98 |
7:  iteration    19700/  173500 | consumed samples:      5043200 | consumed tokens:  10328473600 | elapsed time per iteration (s): 0.37 | learning rate: 1.952E-04 | global batch size:   256 | lm loss: 3.482017E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.808 | TFLOPs: 32.38 |
7:  iteration    19800/  173500 | consumed samples:      5068800 | consumed tokens:  10380902400 | elapsed time per iteration (s): 0.37 | learning rate: 1.951E-04 | global batch size:   256 | lm loss: 3.487890E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.025 | TFLOPs: 32.35 |
7:  iteration    19900/  173500 | consumed samples:      5094400 | consumed tokens:  10433331200 | elapsed time per iteration (s): 0.37 | learning rate: 1.951E-04 | global batch size:   256 | lm loss: 3.484283E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.151 | TFLOPs: 32.35 |
0: [2023-03-17 01:28:00,974] [INFO] [logging.py:68:log_dist] [Rank 0] step=20000, skipped=0, lr=[0.00019502450208460265, 0.00019502450208460265, 0.00019502450208460265], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    20000/  173500 | consumed samples:      5120000 | consumed tokens:  10485760000 | elapsed time per iteration (s): 0.37 | learning rate: 1.950E-04 | global batch size:   256 | lm loss: 3.486893E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.752 | TFLOPs: 32.34 |
0: steps: 20000 loss: 3.4820 iter time (s): 0.368 samples/sec: 696.447
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 20000 | lm loss value: 3.497946E+00 | lm loss PPL: 3.304751E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   20000 to checkpoints_146m91b400m
0: [2023-03-17 01:28:01,106] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step20000 is begin to save!
0: [2023-03-17 01:28:01,135] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_01-model_00-model_states.pt...
0: [2023-03-17 01:28:01,241] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_01-model_00-model_states.pt.
0: [2023-03-17 01:28:01,242] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_03-model_00-model_states.pt...
0: [2023-03-17 01:28:01,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_03-model_00-model_states.pt.
0: [2023-03-17 01:28:01,259] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_04-model_00-model_states.pt...
0: [2023-03-17 01:28:01,274] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_04-model_00-model_states.pt.
0: [2023-03-17 01:28:01,274] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_05-model_00-model_states.pt...
0: [2023-03-17 01:28:01,290] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_05-model_00-model_states.pt.
0: [2023-03-17 01:28:01,290] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_06-model_00-model_states.pt...
0: [2023-03-17 01:28:01,305] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_06-model_00-model_states.pt.
0: [2023-03-17 01:28:01,305] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_07-model_00-model_states.pt...
0: [2023-03-17 01:28:01,320] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_07-model_00-model_states.pt.
0: [2023-03-17 01:28:01,321] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_08-model_00-model_states.pt...
0: [2023-03-17 01:28:01,336] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_08-model_00-model_states.pt.
0: [2023-03-17 01:28:01,336] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_09-model_00-model_states.pt...
0: [2023-03-17 01:28:01,351] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_09-model_00-model_states.pt.
0: [2023-03-17 01:28:01,351] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_10-model_00-model_states.pt...
0: [2023-03-17 01:28:01,366] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_10-model_00-model_states.pt.
0: [2023-03-17 01:28:01,367] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_11-model_00-model_states.pt...
0: [2023-03-17 01:28:01,382] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_11-model_00-model_states.pt.
0: [2023-03-17 01:28:01,382] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_12-model_00-model_states.pt...
0: [2023-03-17 01:28:01,397] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_12-model_00-model_states.pt.
0: [2023-03-17 01:28:01,398] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_13-model_00-model_states.pt...
0: [2023-03-17 01:28:01,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_13-model_00-model_states.pt.
0: [2023-03-17 01:28:01,413] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_14-model_00-model_states.pt...
0: [2023-03-17 01:28:01,428] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_14-model_00-model_states.pt.
0: [2023-03-17 01:28:01,428] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_15-model_00-model_states.pt...
0: [2023-03-17 01:28:01,443] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_15-model_00-model_states.pt.
0: [2023-03-17 01:28:01,444] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_16-model_00-model_states.pt...
0: [2023-03-17 01:28:01,459] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_16-model_00-model_states.pt.
0: [2023-03-17 01:28:01,459] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_17-model_00-model_states.pt...
0: [2023-03-17 01:28:01,474] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_17-model_00-model_states.pt.
0: [2023-03-17 01:28:01,475] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/layer_19-model_00-model_states.pt...
0: [2023-03-17 01:28:01,476] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/layer_19-model_00-model_states.pt.
0: [2023-03-17 01:28:01,476] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step20000/mp_rank_00_model_states.pt
0: [2023-03-17 01:28:01,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/mp_rank_00_model_states.pt...
0: [2023-03-17 01:28:01,479] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/mp_rank_00_model_states.pt.
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
7: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 01:28:01,496] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
0: [2023-03-17 01:28:01,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,533] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,535] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,535] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,536] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,536] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,536] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,536] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,537] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,537] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,537] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,537] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,539] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,539] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,539] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,542] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,542] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,542] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,542] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,542] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,542] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
3: [2023-03-17 01:28:01,542] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 01:28:01,542] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 01:28:01,543] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,543] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,545] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,545] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,545] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,545] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,545] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,545] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 01:28:01,547] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,547] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,547] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,547] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 01:28:01,548] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 01:28:01,548] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 01:28:01,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
6: [2023-03-17 01:28:01,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
2: [2023-03-17 01:28:01,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 01:28:01,552] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,552] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,552] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,552] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,553] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,553] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,553] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,553] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,553] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 01:28:01,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
7: [2023-03-17 01:28:01,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0: [2023-03-17 01:28:01,566] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 01:28:01,566] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
5: [2023-03-17 01:28:01,569] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,571] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
1: [2023-03-17 01:28:01,572] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now!
0:   successfully saved checkpoint at iteration   20000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 478.41
7:  iteration    20100/  173500 | consumed samples:      5145600 | consumed tokens:  10538188800 | elapsed time per iteration (s): 0.38 | learning rate: 1.950E-04 | global batch size:   256 | lm loss: 3.483010E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.512 | TFLOPs: 31.81 |
7:  iteration    20200/  173500 | consumed samples:      5171200 | consumed tokens:  10590617600 | elapsed time per iteration (s): 0.37 | learning rate: 1.949E-04 | global batch size:   256 | lm loss: 3.481857E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.738 | TFLOPs: 32.33 |
7:  iteration    20300/  173500 | consumed samples:      5196800 | consumed tokens:  10643046400 | elapsed time per iteration (s): 0.37 | learning rate: 1.949E-04 | global batch size:   256 | lm loss: 3.478636E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.437 | TFLOPs: 32.32 |
7:  iteration    20400/  173500 | consumed samples:      5222400 | consumed tokens:  10695475200 | elapsed time per iteration (s): 0.37 | learning rate: 1.948E-04 | global batch size:   256 | lm loss: 3.479461E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.934 | TFLOPs: 32.34 |
7:  iteration    20500/  173500 | consumed samples:      5248000 | consumed tokens:  10747904000 | elapsed time per iteration (s): 0.37 | learning rate: 1.948E-04 | global batch size:   256 | lm loss: 3.479193E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.064 | TFLOPs: 32.35 |
7:  iteration    20600/  173500 | consumed samples:      5273600 | consumed tokens:  10800332800 | elapsed time per iteration (s): 0.37 | learning rate: 1.947E-04 | global batch size:   256 | lm loss: 3.478423E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.924 | TFLOPs: 32.02 |
7:  iteration    20700/  173500 | consumed samples:      5299200 | consumed tokens:  10852761600 | elapsed time per iteration (s): 0.37 | learning rate: 1.946E-04 | global batch size:   256 | lm loss: 3.473551E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.490 | TFLOPs: 32.37 |
7:  iteration    20800/  173500 | consumed samples:      5324800 | consumed tokens:  10905190400 | elapsed time per iteration (s): 0.37 | learning rate: 1.946E-04 | global batch size:   256 | lm loss: 3.474691E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.767 | TFLOPs: 32.29 |
7:  iteration    20900/  173500 | consumed samples:      5350400 | consumed tokens:  10957619200 | elapsed time per iteration (s): 0.37 | learning rate: 1.945E-04 | global batch size:   256 | lm loss: 3.475943E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.168 | TFLOPs: 31.93 |
7:  iteration    21000/  173500 | consumed samples:      5376000 | consumed tokens:  11010048000 | elapsed time per iteration (s): 0.38 | learning rate: 1.945E-04 | global batch size:   256 | lm loss: 3.474321E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.437 | TFLOPs: 31.85 |
7:  iteration    21100/  173500 | consumed samples:      5401600 | consumed tokens:  11062476800 | elapsed time per iteration (s): 0.38 | learning rate: 1.944E-04 | global batch size:   256 | lm loss: 3.472135E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.019 | TFLOPs: 31.41 |
7:  iteration    21200/  173500 | consumed samples:      5427200 | consumed tokens:  11114905600 | elapsed time per iteration (s): 0.38 | learning rate: 1.944E-04 | global batch size:   256 | lm loss: 3.469784E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.643 | TFLOPs: 31.86 |
7:  iteration    21300/  173500 | consumed samples:      5452800 | consumed tokens:  11167334400 | elapsed time per iteration (s): 0.38 | learning rate: 1.943E-04 | global batch size:   256 | lm loss: 3.469245E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.493 | TFLOPs: 31.81 |
7:  iteration    21400/  173500 | consumed samples:      5478400 | consumed tokens:  11219763200 | elapsed time per iteration (s): 0.38 | learning rate: 1.942E-04 | global batch size:   256 | lm loss: 3.469918E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.443 | TFLOPs: 31.76 |
7:  iteration    21500/  173500 | consumed samples:      5504000 | consumed tokens:  11272192000 | elapsed time per iteration (s): 0.38 | learning rate: 1.942E-04 | global batch size:   256 | lm loss: 3.471079E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.282 | TFLOPs: 31.43 |
7:  iteration    21600/  173500 | consumed samples:      5529600 | consumed tokens:  11324620800 | elapsed time per iteration (s): 0.38 | learning rate: 1.941E-04 | global batch size:   256 | lm loss: 3.464368E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.043 | TFLOPs: 31.74 |
7:  iteration    21700/  173500 | consumed samples:      5555200 | consumed tokens:  11377049600 | elapsed time per iteration (s): 0.38 | learning rate: 1.941E-04 | global batch size:   256 | lm loss: 3.466189E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.280 | TFLOPs: 31.80 |
7:  iteration    21800/  173500 | consumed samples:      5580800 | consumed tokens:  11429478400 | elapsed time per iteration (s): 0.37 | learning rate: 1.940E-04 | global batch size:   256 | lm loss: 3.463294E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.413 | TFLOPs: 31.95 |
7:  iteration    21900/  173500 | consumed samples:      5606400 | consumed tokens:  11481907200 | elapsed time per iteration (s): 0.37 | learning rate: 1.939E-04 | global batch size:   256 | lm loss: 3.467045E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.500 | TFLOPs: 31.90 |
0: [2023-03-17 01:40:28,954] [INFO] [logging.py:68:log_dist] [Rank 0] step=22000, skipped=0, lr=[0.00019388839136370641, 0.00019388839136370641, 0.00019388839136370641], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    22000/  173500 | consumed samples:      5632000 | consumed tokens:  11534336000 | elapsed time per iteration (s): 0.38 | learning rate: 1.939E-04 | global batch size:   256 | lm loss: 3.463998E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.760 | TFLOPs: 31.78 |
0: steps: 22000 loss: 3.5121 iter time (s): 0.372 samples/sec: 687.705
7:  iteration    22100/  173500 | consumed samples:      5657600 | consumed tokens:  11586764800 | elapsed time per iteration (s): 0.37 | learning rate: 1.938E-04 | global batch size:   256 | lm loss: 3.463206E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.096 | TFLOPs: 32.07 |
7:  iteration    22200/  173500 | consumed samples:      5683200 | consumed tokens:  11639193600 | elapsed time per iteration (s): 0.37 | learning rate: 1.938E-04 | global batch size:   256 | lm loss: 3.466412E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.759 | TFLOPs: 32.10 |
7:  iteration    22300/  173500 | consumed samples:      5708800 | consumed tokens:  11691622400 | elapsed time per iteration (s): 0.37 | learning rate: 1.937E-04 | global batch size:   256 | lm loss: 3.456732E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.994 | TFLOPs: 32.11 |
7:  iteration    22400/  173500 | consumed samples:      5734400 | consumed tokens:  11744051200 | elapsed time per iteration (s): 0.37 | learning rate: 1.936E-04 | global batch size:   256 | lm loss: 3.458852E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.205 | TFLOPs: 32.17 |
7:  iteration    22500/  173500 | consumed samples:      5760000 | consumed tokens:  11796480000 | elapsed time per iteration (s): 0.37 | learning rate: 1.936E-04 | global batch size:   256 | lm loss: 3.457411E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.914 | TFLOPs: 32.20 |
7:  iteration    22600/  173500 | consumed samples:      5785600 | consumed tokens:  11848908800 | elapsed time per iteration (s): 0.37 | learning rate: 1.935E-04 | global batch size:   256 | lm loss: 3.457892E+00 | grad norm: 0.247 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.984 | TFLOPs: 32.11 |
7:  iteration    22700/  173500 | consumed samples:      5811200 | consumed tokens:  11901337600 | elapsed time per iteration (s): 0.37 | learning rate: 1.935E-04 | global batch size:   256 | lm loss: 3.458087E+00 | grad norm: 0.253 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.710 | TFLOPs: 32.15 |
7:  iteration    22800/  173500 | consumed samples:      5836800 | consumed tokens:  11953766400 | elapsed time per iteration (s): 0.37 | learning rate: 1.934E-04 | global batch size:   256 | lm loss: 3.459206E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.812 | TFLOPs: 32.15 |
7:  iteration    22900/  173500 | consumed samples:      5862400 | consumed tokens:  12006195200 | elapsed time per iteration (s): 0.37 | learning rate: 1.933E-04 | global batch size:   256 | lm loss: 3.456480E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.244 | TFLOPs: 32.12 |
7:  iteration    23000/  173500 | consumed samples:      5888000 | consumed tokens:  12058624000 | elapsed time per iteration (s): 0.37 | learning rate: 1.933E-04 | global batch size:   256 | lm loss: 3.451665E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.862 | TFLOPs: 32.06 |
7:  iteration    23100/  173500 | consumed samples:      5913600 | consumed tokens:  12111052800 | elapsed time per iteration (s): 0.37 | learning rate: 1.932E-04 | global batch size:   256 | lm loss: 3.453520E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.832 | TFLOPs: 32.11 |
7:  iteration    23200/  173500 | consumed samples:      5939200 | consumed tokens:  12163481600 | elapsed time per iteration (s): 0.37 | learning rate: 1.932E-04 | global batch size:   256 | lm loss: 3.454173E+00 | grad norm: 0.238 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.892 | TFLOPs: 32.01 |
7:  iteration    23300/  173500 | consumed samples:      5964800 | consumed tokens:  12215910400 | elapsed time per iteration (s): 0.37 | learning rate: 1.931E-04 | global batch size:   256 | lm loss: 3.452790E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.220 | TFLOPs: 31.94 |
7:  iteration    23400/  173500 | consumed samples:      5990400 | consumed tokens:  12268339200 | elapsed time per iteration (s): 0.37 | learning rate: 1.930E-04 | global batch size:   256 | lm loss: 3.455974E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.416 | TFLOPs: 31.95 |
7:  iteration    23500/  173500 | consumed samples:      6016000 | consumed tokens:  12320768000 | elapsed time per iteration (s): 0.37 | learning rate: 1.930E-04 | global batch size:   256 | lm loss: 3.451365E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.069 | TFLOPs: 32.16 |
7:  iteration    23600/  173500 | consumed samples:      6041600 | consumed tokens:  12373196800 | elapsed time per iteration (s): 0.37 | learning rate: 1.929E-04 | global batch size:   256 | lm loss: 3.451422E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.078 | TFLOPs: 32.12 |
7:  iteration    23700/  173500 | consumed samples:      6067200 | consumed tokens:  12425625600 | elapsed time per iteration (s): 0.37 | learning rate: 1.928E-04 | global batch size:   256 | lm loss: 3.447089E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.878 | TFLOPs: 32.11 |
7:  iteration    23800/  173500 | consumed samples:      6092800 | consumed tokens:  12478054400 | elapsed time per iteration (s): 0.37 | learning rate: 1.928E-04 | global batch size:   256 | lm loss: 3.448462E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.591 | TFLOPs: 32.14 |
7:  iteration    23900/  173500 | consumed samples:      6118400 | consumed tokens:  12530483200 | elapsed time per iteration (s): 0.37 | learning rate: 1.927E-04 | global batch size:   256 | lm loss: 3.450605E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.425 | TFLOPs: 32.09 |
0: [2023-03-17 01:52:53,392] [INFO] [logging.py:68:log_dist] [Rank 0] step=24000, skipped=0, lr=[0.00019264004235759096, 0.00019264004235759096, 0.00019264004235759096], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    24000/  173500 | consumed samples:      6144000 | consumed tokens:  12582912000 | elapsed time per iteration (s): 0.37 | learning rate: 1.926E-04 | global batch size:   256 | lm loss: 3.447266E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.405 | TFLOPs: 32.18 |
0: steps: 24000 loss: 3.4306 iter time (s): 0.370 samples/sec: 691.399
7:  iteration    24100/  173500 | consumed samples:      6169600 | consumed tokens:  12635340800 | elapsed time per iteration (s): 0.37 | learning rate: 1.926E-04 | global batch size:   256 | lm loss: 3.445032E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.399 | TFLOPs: 31.95 |
7:  iteration    24200/  173500 | consumed samples:      6195200 | consumed tokens:  12687769600 | elapsed time per iteration (s): 0.37 | learning rate: 1.925E-04 | global batch size:   256 | lm loss: 3.444727E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.524 | TFLOPs: 32.14 |
7:  iteration    24300/  173500 | consumed samples:      6220800 | consumed tokens:  12740198400 | elapsed time per iteration (s): 0.37 | learning rate: 1.924E-04 | global batch size:   256 | lm loss: 3.440475E+00 | grad norm: 0.269 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.987 | TFLOPs: 32.25 |
7:  iteration    24400/  173500 | consumed samples:      6246400 | consumed tokens:  12792627200 | elapsed time per iteration (s): 0.37 | learning rate: 1.924E-04 | global batch size:   256 | lm loss: 3.442089E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.230 | TFLOPs: 32.36 |
7:  iteration    24500/  173500 | consumed samples:      6272000 | consumed tokens:  12845056000 | elapsed time per iteration (s): 0.37 | learning rate: 1.923E-04 | global batch size:   256 | lm loss: 3.443986E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.007 | TFLOPs: 32.39 |
7:  iteration    24600/  173500 | consumed samples:      6297600 | consumed tokens:  12897484800 | elapsed time per iteration (s): 0.37 | learning rate: 1.922E-04 | global batch size:   256 | lm loss: 3.447919E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.025 | TFLOPs: 32.30 |
7:  iteration    24700/  173500 | consumed samples:      6323200 | consumed tokens:  12949913600 | elapsed time per iteration (s): 0.37 | learning rate: 1.922E-04 | global batch size:   256 | lm loss: 3.440903E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.237 | TFLOPs: 32.31 |
7:  iteration    24800/  173500 | consumed samples:      6348800 | consumed tokens:  13002342400 | elapsed time per iteration (s): 0.37 | learning rate: 1.921E-04 | global batch size:   256 | lm loss: 3.443869E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.598 | TFLOPs: 32.19 |
7:  iteration    24900/  173500 | consumed samples:      6374400 | consumed tokens:  13054771200 | elapsed time per iteration (s): 0.37 | learning rate: 1.920E-04 | global batch size:   256 | lm loss: 3.440027E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.597 | TFLOPs: 32.37 |
7:  iteration    25000/  173500 | consumed samples:      6400000 | consumed tokens:  13107200000 | elapsed time per iteration (s): 0.37 | learning rate: 1.920E-04 | global batch size:   256 | lm loss: 3.437885E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.558 | TFLOPs: 32.37 |
7:  iteration    25100/  173500 | consumed samples:      6425600 | consumed tokens:  13159628800 | elapsed time per iteration (s): 0.37 | learning rate: 1.919E-04 | global batch size:   256 | lm loss: 3.439690E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.327 | TFLOPs: 32.36 |
7:  iteration    25200/  173500 | consumed samples:      6451200 | consumed tokens:  13212057600 | elapsed time per iteration (s): 0.37 | learning rate: 1.918E-04 | global batch size:   256 | lm loss: 3.433900E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.830 | TFLOPs: 32.39 |
7:  iteration    25300/  173500 | consumed samples:      6476800 | consumed tokens:  13264486400 | elapsed time per iteration (s): 0.37 | learning rate: 1.918E-04 | global batch size:   256 | lm loss: 3.434483E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.143 | TFLOPs: 32.35 |
7:  iteration    25400/  173500 | consumed samples:      6502400 | consumed tokens:  13316915200 | elapsed time per iteration (s): 0.37 | learning rate: 1.917E-04 | global batch size:   256 | lm loss: 3.440547E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.121 | TFLOPs: 32.35 |
7:  iteration    25500/  173500 | consumed samples:      6528000 | consumed tokens:  13369344000 | elapsed time per iteration (s): 0.37 | learning rate: 1.916E-04 | global batch size:   256 | lm loss: 3.434506E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.711 | TFLOPs: 32.29 |
7:  iteration    25600/  173500 | consumed samples:      6553600 | consumed tokens:  13421772800 | elapsed time per iteration (s): 0.37 | learning rate: 1.916E-04 | global batch size:   256 | lm loss: 3.436456E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.357 | TFLOPs: 32.36 |
7:  iteration    25700/  173500 | consumed samples:      6579200 | consumed tokens:  13474201600 | elapsed time per iteration (s): 0.37 | learning rate: 1.915E-04 | global batch size:   256 | lm loss: 3.434398E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.523 | TFLOPs: 32.32 |
7:  iteration    25800/  173500 | consumed samples:      6604800 | consumed tokens:  13526630400 | elapsed time per iteration (s): 0.37 | learning rate: 1.914E-04 | global batch size:   256 | lm loss: 3.431340E+00 | grad norm: 0.252 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.236 | TFLOPs: 32.36 |
7:  iteration    25900/  173500 | consumed samples:      6630400 | consumed tokens:  13579059200 | elapsed time per iteration (s): 0.37 | learning rate: 1.914E-04 | global batch size:   256 | lm loss: 3.435319E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.380 | TFLOPs: 32.36 |
0: [2023-03-17 02:05:13,161] [INFO] [logging.py:68:log_dist] [Rank 0] step=26000, skipped=0, lr=[0.00019128112529201118, 0.00019128112529201118, 0.00019128112529201118], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    26000/  173500 | consumed samples:      6656000 | consumed tokens:  13631488000 | elapsed time per iteration (s): 0.37 | learning rate: 1.913E-04 | global batch size:   256 | lm loss: 3.432669E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.519 | TFLOPs: 32.32 |
0: steps: 26000 loss: 3.3961 iter time (s): 0.368 samples/sec: 695.865
7:  iteration    26100/  173500 | consumed samples:      6681600 | consumed tokens:  13683916800 | elapsed time per iteration (s): 0.37 | learning rate: 1.912E-04 | global batch size:   256 | lm loss: 3.435576E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.680 | TFLOPs: 32.33 |
7:  iteration    26200/  173500 | consumed samples:      6707200 | consumed tokens:  13736345600 | elapsed time per iteration (s): 0.37 | learning rate: 1.911E-04 | global batch size:   256 | lm loss: 3.432892E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.497 | TFLOPs: 32.37 |
7:  iteration    26300/  173500 | consumed samples:      6732800 | consumed tokens:  13788774400 | elapsed time per iteration (s): 0.37 | learning rate: 1.911E-04 | global batch size:   256 | lm loss: 3.436319E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.285 | TFLOPs: 32.36 |
7:  iteration    26400/  173500 | consumed samples:      6758400 | consumed tokens:  13841203200 | elapsed time per iteration (s): 0.37 | learning rate: 1.910E-04 | global batch size:   256 | lm loss: 3.432904E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.959 | TFLOPs: 32.39 |
7:  iteration    26500/  173500 | consumed samples:      6784000 | consumed tokens:  13893632000 | elapsed time per iteration (s): 0.37 | learning rate: 1.909E-04 | global batch size:   256 | lm loss: 3.429234E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.514 | TFLOPs: 32.37 |
7:  iteration    26600/  173500 | consumed samples:      6809600 | consumed tokens:  13946060800 | elapsed time per iteration (s): 0.37 | learning rate: 1.909E-04 | global batch size:   256 | lm loss: 3.432322E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.594 | TFLOPs: 32.33 |
7:  iteration    26700/  173500 | consumed samples:      6835200 | consumed tokens:  13998489600 | elapsed time per iteration (s): 0.37 | learning rate: 1.908E-04 | global batch size:   256 | lm loss: 3.431240E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.095 | TFLOPs: 32.40 |
7:  iteration    26800/  173500 | consumed samples:      6860800 | consumed tokens:  14050918400 | elapsed time per iteration (s): 0.37 | learning rate: 1.907E-04 | global batch size:   256 | lm loss: 3.429087E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.072 | TFLOPs: 32.40 |
7:  iteration    26900/  173500 | consumed samples:      6886400 | consumed tokens:  14103347200 | elapsed time per iteration (s): 0.37 | learning rate: 1.906E-04 | global batch size:   256 | lm loss: 3.426969E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.871 | TFLOPs: 32.39 |
7:  iteration    27000/  173500 | consumed samples:      6912000 | consumed tokens:  14155776000 | elapsed time per iteration (s): 0.37 | learning rate: 1.906E-04 | global batch size:   256 | lm loss: 3.426473E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.304 | TFLOPs: 32.36 |
7:  iteration    27100/  173500 | consumed samples:      6937600 | consumed tokens:  14208204800 | elapsed time per iteration (s): 0.37 | learning rate: 1.905E-04 | global batch size:   256 | lm loss: 3.429912E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.967 | TFLOPs: 32.39 |
7:  iteration    27200/  173500 | consumed samples:      6963200 | consumed tokens:  14260633600 | elapsed time per iteration (s): 0.37 | learning rate: 1.904E-04 | global batch size:   256 | lm loss: 3.428336E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.356 | TFLOPs: 32.36 |
7:  iteration    27300/  173500 | consumed samples:      6988800 | consumed tokens:  14313062400 | elapsed time per iteration (s): 0.37 | learning rate: 1.903E-04 | global batch size:   256 | lm loss: 3.428675E+00 | grad norm: 0.258 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.401 | TFLOPs: 32.37 |
7:  iteration    27400/  173500 | consumed samples:      7014400 | consumed tokens:  14365491200 | elapsed time per iteration (s): 0.37 | learning rate: 1.903E-04 | global batch size:   256 | lm loss: 3.424103E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.548 | TFLOPs: 32.37 |
7:  iteration    27500/  173500 | consumed samples:      7040000 | consumed tokens:  14417920000 | elapsed time per iteration (s): 0.37 | learning rate: 1.902E-04 | global batch size:   256 | lm loss: 3.420227E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.894 | TFLOPs: 32.39 |
7:  iteration    27600/  173500 | consumed samples:      7065600 | consumed tokens:  14470348800 | elapsed time per iteration (s): 0.37 | learning rate: 1.901E-04 | global batch size:   256 | lm loss: 3.428957E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.901 | TFLOPs: 32.39 |
7:  iteration    27700/  173500 | consumed samples:      7091200 | consumed tokens:  14522777600 | elapsed time per iteration (s): 0.37 | learning rate: 1.900E-04 | global batch size:   256 | lm loss: 3.427399E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.472 | TFLOPs: 32.37 |
7:  iteration    27800/  173500 | consumed samples:      7116800 | consumed tokens:  14575206400 | elapsed time per iteration (s): 0.37 | learning rate: 1.900E-04 | global batch size:   256 | lm loss: 3.422796E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.303 | TFLOPs: 32.36 |
7:  iteration    27900/  173500 | consumed samples:      7142400 | consumed tokens:  14627635200 | elapsed time per iteration (s): 0.37 | learning rate: 1.899E-04 | global batch size:   256 | lm loss: 3.422589E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.854 | TFLOPs: 32.34 |
0: [2023-03-17 02:17:31,512] [INFO] [logging.py:68:log_dist] [Rank 0] step=28000, skipped=0, lr=[0.00018981345832700956, 0.00018981345832700956, 0.00018981345832700956], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    28000/  173500 | consumed samples:      7168000 | consumed tokens:  14680064000 | elapsed time per iteration (s): 0.37 | learning rate: 1.898E-04 | global batch size:   256 | lm loss: 3.422470E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.176 | TFLOPs: 32.31 |
0: steps: 28000 loss: 3.4765 iter time (s): 0.367 samples/sec: 697.327
7:  iteration    28100/  173500 | consumed samples:      7193600 | consumed tokens:  14732492800 | elapsed time per iteration (s): 0.37 | learning rate: 1.897E-04 | global batch size:   256 | lm loss: 3.421062E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.742 | TFLOPs: 32.33 |
7:  iteration    28200/  173500 | consumed samples:      7219200 | consumed tokens:  14784921600 | elapsed time per iteration (s): 0.37 | learning rate: 1.897E-04 | global batch size:   256 | lm loss: 3.422597E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.677 | TFLOPs: 32.38 |
7:  iteration    28300/  173500 | consumed samples:      7244800 | consumed tokens:  14837350400 | elapsed time per iteration (s): 0.37 | learning rate: 1.896E-04 | global batch size:   256 | lm loss: 3.417150E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.990 | TFLOPs: 32.35 |
7:  iteration    28400/  173500 | consumed samples:      7270400 | consumed tokens:  14889779200 | elapsed time per iteration (s): 0.37 | learning rate: 1.895E-04 | global batch size:   256 | lm loss: 3.423674E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.614 | TFLOPs: 32.38 |
7:  iteration    28500/  173500 | consumed samples:      7296000 | consumed tokens:  14942208000 | elapsed time per iteration (s): 0.37 | learning rate: 1.894E-04 | global batch size:   256 | lm loss: 3.420557E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.032 | TFLOPs: 32.35 |
7:  iteration    28600/  173500 | consumed samples:      7321600 | consumed tokens:  14994636800 | elapsed time per iteration (s): 0.37 | learning rate: 1.894E-04 | global batch size:   256 | lm loss: 3.418424E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.726 | TFLOPs: 32.33 |
7:  iteration    28700/  173500 | consumed samples:      7347200 | consumed tokens:  15047065600 | elapsed time per iteration (s): 0.37 | learning rate: 1.893E-04 | global batch size:   256 | lm loss: 3.416770E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.759 | TFLOPs: 32.34 |
7:  iteration    28800/  173500 | consumed samples:      7372800 | consumed tokens:  15099494400 | elapsed time per iteration (s): 0.37 | learning rate: 1.892E-04 | global batch size:   256 | lm loss: 3.423238E+00 | grad norm: 0.257 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.827 | TFLOPs: 32.34 |
7:  iteration    28900/  173500 | consumed samples:      7398400 | consumed tokens:  15151923200 | elapsed time per iteration (s): 0.37 | learning rate: 1.891E-04 | global batch size:   256 | lm loss: 3.417028E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.054 | TFLOPs: 32.35 |
7:  iteration    29000/  173500 | consumed samples:      7424000 | consumed tokens:  15204352000 | elapsed time per iteration (s): 0.37 | learning rate: 1.890E-04 | global batch size:   256 | lm loss: 3.417631E+00 | grad norm: 0.248 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.353 | TFLOPs: 32.36 |
7:  iteration    29100/  173500 | consumed samples:      7449600 | consumed tokens:  15256780800 | elapsed time per iteration (s): 0.37 | learning rate: 1.890E-04 | global batch size:   256 | lm loss: 3.417578E+00 | grad norm: 0.254 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.838 | TFLOPs: 32.29 |
7:  iteration    29200/  173500 | consumed samples:      7475200 | consumed tokens:  15309209600 | elapsed time per iteration (s): 0.37 | learning rate: 1.889E-04 | global batch size:   256 | lm loss: 3.414700E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.147 | TFLOPs: 32.26 |
7:  iteration    29300/  173500 | consumed samples:      7500800 | consumed tokens:  15361638400 | elapsed time per iteration (s): 0.37 | learning rate: 1.888E-04 | global batch size:   256 | lm loss: 3.411299E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.636 | TFLOPs: 32.24 |
7:  iteration    29400/  173500 | consumed samples:      7526400 | consumed tokens:  15414067200 | elapsed time per iteration (s): 0.37 | learning rate: 1.887E-04 | global batch size:   256 | lm loss: 3.416735E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.170 | TFLOPs: 32.26 |
7:  iteration    29500/  173500 | consumed samples:      7552000 | consumed tokens:  15466496000 | elapsed time per iteration (s): 0.37 | learning rate: 1.886E-04 | global batch size:   256 | lm loss: 3.410532E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.177 | TFLOPs: 32.35 |
7:  iteration    29600/  173500 | consumed samples:      7577600 | consumed tokens:  15518924800 | elapsed time per iteration (s): 0.37 | learning rate: 1.886E-04 | global batch size:   256 | lm loss: 3.416582E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.971 | TFLOPs: 32.35 |
7:  iteration    29700/  173500 | consumed samples:      7603200 | consumed tokens:  15571353600 | elapsed time per iteration (s): 0.37 | learning rate: 1.885E-04 | global batch size:   256 | lm loss: 3.409717E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.667 | TFLOPs: 32.33 |
7:  iteration    29800/  173500 | consumed samples:      7628800 | consumed tokens:  15623782400 | elapsed time per iteration (s): 0.37 | learning rate: 1.884E-04 | global batch size:   256 | lm loss: 3.415025E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.573 | TFLOPs: 32.28 |
7:  iteration    29900/  173500 | consumed samples:      7654400 | consumed tokens:  15676211200 | elapsed time per iteration (s): 0.37 | learning rate: 1.883E-04 | global batch size:   256 | lm loss: 3.408424E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.422 | TFLOPs: 32.27 |
0: [2023-03-17 02:29:50,959] [INFO] [logging.py:68:log_dist] [Rank 0] step=30000, skipped=0, lr=[0.00018823900512431258, 0.00018823900512431258, 0.00018823900512431258], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    30000/  173500 | consumed samples:      7680000 | consumed tokens:  15728640000 | elapsed time per iteration (s): 0.37 | learning rate: 1.882E-04 | global batch size:   256 | lm loss: 3.412630E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.858 | TFLOPs: 32.25 |
0: steps: 30000 loss: 3.4140 iter time (s): 0.368 samples/sec: 696.261
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 30000 | lm loss value: 3.443433E+00 | lm loss PPL: 3.129422E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   30000 to checkpoints_146m91b400m
0: [2023-03-17 02:29:51,085] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step30000 is begin to save!
0: [2023-03-17 02:29:51,088] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_01-model_00-model_states.pt...
0: [2023-03-17 02:29:51,179] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_01-model_00-model_states.pt.
0: [2023-03-17 02:29:51,180] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_03-model_00-model_states.pt...
0: [2023-03-17 02:29:51,195] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_03-model_00-model_states.pt.
0: [2023-03-17 02:29:51,196] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_04-model_00-model_states.pt...
0: [2023-03-17 02:29:51,211] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_04-model_00-model_states.pt.
0: [2023-03-17 02:29:51,211] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_05-model_00-model_states.pt...
0: [2023-03-17 02:29:51,226] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_05-model_00-model_states.pt.
0: [2023-03-17 02:29:51,226] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_06-model_00-model_states.pt...
0: [2023-03-17 02:29:51,241] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_06-model_00-model_states.pt.
0: [2023-03-17 02:29:51,241] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_07-model_00-model_states.pt...
0: [2023-03-17 02:29:51,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_07-model_00-model_states.pt.
0: [2023-03-17 02:29:51,256] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_08-model_00-model_states.pt...
0: [2023-03-17 02:29:51,271] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_08-model_00-model_states.pt.
0: [2023-03-17 02:29:51,271] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_09-model_00-model_states.pt...
0: [2023-03-17 02:29:51,286] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_09-model_00-model_states.pt.
0: [2023-03-17 02:29:51,286] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_10-model_00-model_states.pt...
0: [2023-03-17 02:29:51,301] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_10-model_00-model_states.pt.
0: [2023-03-17 02:29:51,302] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_11-model_00-model_states.pt...
0: [2023-03-17 02:29:51,316] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_11-model_00-model_states.pt.
0: [2023-03-17 02:29:51,317] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_12-model_00-model_states.pt...
0: [2023-03-17 02:29:51,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_12-model_00-model_states.pt.
0: [2023-03-17 02:29:51,332] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_13-model_00-model_states.pt...
0: [2023-03-17 02:29:51,347] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_13-model_00-model_states.pt.
0: [2023-03-17 02:29:51,347] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_14-model_00-model_states.pt...
0: [2023-03-17 02:29:51,362] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_14-model_00-model_states.pt.
0: [2023-03-17 02:29:51,362] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_15-model_00-model_states.pt...
0: [2023-03-17 02:29:51,377] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_15-model_00-model_states.pt.
0: [2023-03-17 02:29:51,377] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_16-model_00-model_states.pt...
0: [2023-03-17 02:29:51,392] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_16-model_00-model_states.pt.
0: [2023-03-17 02:29:51,392] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_17-model_00-model_states.pt...
0: [2023-03-17 02:29:51,407] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_17-model_00-model_states.pt.
0: [2023-03-17 02:29:51,408] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/layer_19-model_00-model_states.pt...
0: [2023-03-17 02:29:51,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/layer_19-model_00-model_states.pt.
0: [2023-03-17 02:29:51,409] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step30000/mp_rank_00_model_states.pt
0: [2023-03-17 02:29:51,409] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/mp_rank_00_model_states.pt...
0: [2023-03-17 02:29:51,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/mp_rank_00_model_states.pt.
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 02:29:51,430] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
0: [2023-03-17 02:29:51,463] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,468] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,468] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,468] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,468] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,468] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,468] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,469] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,469] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,469] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,469] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,469] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,471] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 02:29:51,471] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,471] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0: [2023-03-17 02:29:51,492] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 02:29:51,492] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
2: [2023-03-17 02:29:51,497] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
1: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
6: [2023-03-17 02:29:51,501] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
4: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
7: [2023-03-17 02:29:51,506] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
3: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
5: [2023-03-17 02:29:51,507] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now!
0:   successfully saved checkpoint at iteration   30000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 429.21
7:  iteration    30100/  173500 | consumed samples:      7705600 | consumed tokens:  15781068800 | elapsed time per iteration (s): 0.37 | learning rate: 1.882E-04 | global batch size:   256 | lm loss: 3.413317E+00 | grad norm: 0.248 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.859 | TFLOPs: 31.87 |
7:  iteration    30200/  173500 | consumed samples:      7731200 | consumed tokens:  15833497600 | elapsed time per iteration (s): 0.37 | learning rate: 1.881E-04 | global batch size:   256 | lm loss: 3.412453E+00 | grad norm: 0.261 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.041 | TFLOPs: 32.30 |
7:  iteration    30300/  173500 | consumed samples:      7756800 | consumed tokens:  15885926400 | elapsed time per iteration (s): 0.37 | learning rate: 1.880E-04 | global batch size:   256 | lm loss: 3.411611E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.550 | TFLOPs: 32.37 |
7:  iteration    30400/  173500 | consumed samples:      7782400 | consumed tokens:  15938355200 | elapsed time per iteration (s): 0.37 | learning rate: 1.879E-04 | global batch size:   256 | lm loss: 3.413736E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.485 | TFLOPs: 32.37 |
7:  iteration    30500/  173500 | consumed samples:      7808000 | consumed tokens:  15990784000 | elapsed time per iteration (s): 0.37 | learning rate: 1.878E-04 | global batch size:   256 | lm loss: 3.405335E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.261 | TFLOPs: 32.36 |
7:  iteration    30600/  173500 | consumed samples:      7833600 | consumed tokens:  16043212800 | elapsed time per iteration (s): 0.37 | learning rate: 1.877E-04 | global batch size:   256 | lm loss: 3.410105E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.515 | TFLOPs: 32.14 |
7:  iteration    30700/  173500 | consumed samples:      7859200 | consumed tokens:  16095641600 | elapsed time per iteration (s): 0.37 | learning rate: 1.877E-04 | global batch size:   256 | lm loss: 3.410781E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.183 | TFLOPs: 32.31 |
7:  iteration    30800/  173500 | consumed samples:      7884800 | consumed tokens:  16148070400 | elapsed time per iteration (s): 0.37 | learning rate: 1.876E-04 | global batch size:   256 | lm loss: 3.404511E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.481 | TFLOPs: 32.37 |
7:  iteration    30900/  173500 | consumed samples:      7910400 | consumed tokens:  16200499200 | elapsed time per iteration (s): 0.37 | learning rate: 1.875E-04 | global batch size:   256 | lm loss: 3.405917E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.503 | TFLOPs: 32.37 |
7:  iteration    31000/  173500 | consumed samples:      7936000 | consumed tokens:  16252928000 | elapsed time per iteration (s): 0.37 | learning rate: 1.874E-04 | global batch size:   256 | lm loss: 3.408833E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.178 | TFLOPs: 32.36 |
7:  iteration    31100/  173500 | consumed samples:      7961600 | consumed tokens:  16305356800 | elapsed time per iteration (s): 0.37 | learning rate: 1.873E-04 | global batch size:   256 | lm loss: 3.405701E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.976 | TFLOPs: 32.30 |
7:  iteration    31200/  173500 | consumed samples:      7987200 | consumed tokens:  16357785600 | elapsed time per iteration (s): 0.37 | learning rate: 1.872E-04 | global batch size:   256 | lm loss: 3.404145E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.554 | TFLOPs: 32.37 |
7:  iteration    31300/  173500 | consumed samples:      8012800 | consumed tokens:  16410214400 | elapsed time per iteration (s): 0.37 | learning rate: 1.872E-04 | global batch size:   256 | lm loss: 3.403844E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.701 | TFLOPs: 32.38 |
7:  iteration    31400/  173500 | consumed samples:      8038400 | consumed tokens:  16462643200 | elapsed time per iteration (s): 0.37 | learning rate: 1.871E-04 | global batch size:   256 | lm loss: 3.405614E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.947 | TFLOPs: 32.11 |
7:  iteration    31500/  173500 | consumed samples:      8064000 | consumed tokens:  16515072000 | elapsed time per iteration (s): 0.38 | learning rate: 1.870E-04 | global batch size:   256 | lm loss: 3.402699E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.543 | TFLOPs: 31.72 |
7:  iteration    31600/  173500 | consumed samples:      8089600 | consumed tokens:  16567500800 | elapsed time per iteration (s): 0.38 | learning rate: 1.869E-04 | global batch size:   256 | lm loss: 3.410608E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.745 | TFLOPs: 31.68 |
7:  iteration    31700/  173500 | consumed samples:      8115200 | consumed tokens:  16619929600 | elapsed time per iteration (s): 0.37 | learning rate: 1.868E-04 | global batch size:   256 | lm loss: 3.406237E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.446 | TFLOPs: 31.99 |
7:  iteration    31800/  173500 | consumed samples:      8140800 | consumed tokens:  16672358400 | elapsed time per iteration (s): 0.37 | learning rate: 1.867E-04 | global batch size:   256 | lm loss: 3.401314E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.062 | TFLOPs: 32.16 |
7:  iteration    31900/  173500 | consumed samples:      8166400 | consumed tokens:  16724787200 | elapsed time per iteration (s): 0.37 | learning rate: 1.866E-04 | global batch size:   256 | lm loss: 3.400880E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.517 | TFLOPs: 31.95 |
0: [2023-03-17 02:42:13,823] [INFO] [logging.py:68:log_dist] [Rank 0] step=32000, skipped=0, lr=[0.00018655987222005428, 0.00018655987222005428, 0.00018655987222005428], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    32000/  173500 | consumed samples:      8192000 | consumed tokens:  16777216000 | elapsed time per iteration (s): 0.37 | learning rate: 1.866E-04 | global batch size:   256 | lm loss: 3.401601E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.418 | TFLOPs: 31.95 |
0: steps: 32000 loss: 3.3827 iter time (s): 0.370 samples/sec: 692.356
7:  iteration    32100/  173500 | consumed samples:      8217600 | consumed tokens:  16829644800 | elapsed time per iteration (s): 0.37 | learning rate: 1.865E-04 | global batch size:   256 | lm loss: 3.403542E+00 | grad norm: 0.193 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.399 | TFLOPs: 32.09 |
7:  iteration    32200/  173500 | consumed samples:      8243200 | consumed tokens:  16882073600 | elapsed time per iteration (s): 0.37 | learning rate: 1.864E-04 | global batch size:   256 | lm loss: 3.403383E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.148 | TFLOPs: 32.07 |
7:  iteration    32300/  173500 | consumed samples:      8268800 | consumed tokens:  16934502400 | elapsed time per iteration (s): 0.37 | learning rate: 1.863E-04 | global batch size:   256 | lm loss: 3.406075E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.497 | TFLOPs: 32.00 |
7:  iteration    32400/  173500 | consumed samples:      8294400 | consumed tokens:  16986931200 | elapsed time per iteration (s): 0.37 | learning rate: 1.862E-04 | global batch size:   256 | lm loss: 3.401762E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.622 | TFLOPs: 31.96 |
7:  iteration    32500/  173500 | consumed samples:      8320000 | consumed tokens:  17039360000 | elapsed time per iteration (s): 0.37 | learning rate: 1.861E-04 | global batch size:   256 | lm loss: 3.400850E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.126 | TFLOPs: 32.03 |
7:  iteration    32600/  173500 | consumed samples:      8345600 | consumed tokens:  17091788800 | elapsed time per iteration (s): 0.39 | learning rate: 1.860E-04 | global batch size:   256 | lm loss: 3.398781E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 659.514 | TFLOPs: 30.78 |
7:  iteration    32700/  173500 | consumed samples:      8371200 | consumed tokens:  17144217600 | elapsed time per iteration (s): 0.38 | learning rate: 1.859E-04 | global batch size:   256 | lm loss: 3.400369E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 666.605 | TFLOPs: 31.11 |
7:  iteration    32800/  173500 | consumed samples:      8396800 | consumed tokens:  17196646400 | elapsed time per iteration (s): 0.38 | learning rate: 1.859E-04 | global batch size:   256 | lm loss: 3.398832E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 671.831 | TFLOPs: 31.36 |
7:  iteration    32900/  173500 | consumed samples:      8422400 | consumed tokens:  17249075200 | elapsed time per iteration (s): 0.38 | learning rate: 1.858E-04 | global batch size:   256 | lm loss: 3.398994E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 669.348 | TFLOPs: 31.24 |
7:  iteration    33000/  173500 | consumed samples:      8448000 | consumed tokens:  17301504000 | elapsed time per iteration (s): 0.39 | learning rate: 1.857E-04 | global batch size:   256 | lm loss: 3.401393E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 664.064 | TFLOPs: 31.00 |
7:  iteration    33100/  173500 | consumed samples:      8473600 | consumed tokens:  17353932800 | elapsed time per iteration (s): 0.38 | learning rate: 1.856E-04 | global batch size:   256 | lm loss: 3.395030E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.421 | TFLOPs: 31.39 |
7:  iteration    33200/  173500 | consumed samples:      8499200 | consumed tokens:  17406361600 | elapsed time per iteration (s): 0.38 | learning rate: 1.855E-04 | global batch size:   256 | lm loss: 3.393362E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.109 | TFLOPs: 31.42 |
7:  iteration    33300/  173500 | consumed samples:      8524800 | consumed tokens:  17458790400 | elapsed time per iteration (s): 0.38 | learning rate: 1.854E-04 | global batch size:   256 | lm loss: 3.398211E+00 | grad norm: 0.256 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 675.890 | TFLOPs: 31.55 |
7:  iteration    33400/  173500 | consumed samples:      8550400 | consumed tokens:  17511219200 | elapsed time per iteration (s): 0.38 | learning rate: 1.853E-04 | global batch size:   256 | lm loss: 3.397445E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 668.454 | TFLOPs: 31.20 |
7:  iteration    33500/  173500 | consumed samples:      8576000 | consumed tokens:  17563648000 | elapsed time per iteration (s): 0.38 | learning rate: 1.852E-04 | global batch size:   256 | lm loss: 3.398410E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.951 | TFLOPs: 31.69 |
7:  iteration    33600/  173500 | consumed samples:      8601600 | consumed tokens:  17616076800 | elapsed time per iteration (s): 0.37 | learning rate: 1.851E-04 | global batch size:   256 | lm loss: 3.395610E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.190 | TFLOPs: 31.94 |
7:  iteration    33700/  173500 | consumed samples:      8627200 | consumed tokens:  17668505600 | elapsed time per iteration (s): 0.37 | learning rate: 1.851E-04 | global batch size:   256 | lm loss: 3.397402E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.195 | TFLOPs: 31.94 |
7:  iteration    33800/  173500 | consumed samples:      8652800 | consumed tokens:  17720934400 | elapsed time per iteration (s): 0.37 | learning rate: 1.850E-04 | global batch size:   256 | lm loss: 3.395449E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.954 | TFLOPs: 32.06 |
7:  iteration    33900/  173500 | consumed samples:      8678400 | consumed tokens:  17773363200 | elapsed time per iteration (s): 0.37 | learning rate: 1.849E-04 | global batch size:   256 | lm loss: 3.397175E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.522 | TFLOPs: 32.00 |
0: [2023-03-17 02:54:49,243] [INFO] [logging.py:68:log_dist] [Rank 0] step=34000, skipped=0, lr=[0.00018477830620634072, 0.00018477830620634072, 0.00018477830620634072], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    34000/  173500 | consumed samples:      8704000 | consumed tokens:  17825792000 | elapsed time per iteration (s): 0.37 | learning rate: 1.848E-04 | global batch size:   256 | lm loss: 3.392925E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.054 | TFLOPs: 32.02 |
0: steps: 34000 loss: 3.3724 iter time (s): 0.376 samples/sec: 681.120
7:  iteration    34100/  173500 | consumed samples:      8729600 | consumed tokens:  17878220800 | elapsed time per iteration (s): 0.37 | learning rate: 1.847E-04 | global batch size:   256 | lm loss: 3.397301E+00 | grad norm: 0.270 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.378 | TFLOPs: 32.08 |
7:  iteration    34200/  173500 | consumed samples:      8755200 | consumed tokens:  17930649600 | elapsed time per iteration (s): 0.37 | learning rate: 1.846E-04 | global batch size:   256 | lm loss: 3.391086E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.973 | TFLOPs: 31.97 |
7:  iteration    34300/  173500 | consumed samples:      8780800 | consumed tokens:  17983078400 | elapsed time per iteration (s): 0.37 | learning rate: 1.845E-04 | global batch size:   256 | lm loss: 3.394380E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.730 | TFLOPs: 32.05 |
7:  iteration    34400/  173500 | consumed samples:      8806400 | consumed tokens:  18035507200 | elapsed time per iteration (s): 0.37 | learning rate: 1.844E-04 | global batch size:   256 | lm loss: 3.395203E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.402 | TFLOPs: 32.18 |
7:  iteration    34500/  173500 | consumed samples:      8832000 | consumed tokens:  18087936000 | elapsed time per iteration (s): 0.37 | learning rate: 1.843E-04 | global batch size:   256 | lm loss: 3.390618E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.558 | TFLOPs: 32.23 |
7:  iteration    34600/  173500 | consumed samples:      8857600 | consumed tokens:  18140364800 | elapsed time per iteration (s): 0.37 | learning rate: 1.842E-04 | global batch size:   256 | lm loss: 3.393254E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.007 | TFLOPs: 31.97 |
7:  iteration    34700/  173500 | consumed samples:      8883200 | consumed tokens:  18192793600 | elapsed time per iteration (s): 0.37 | learning rate: 1.841E-04 | global batch size:   256 | lm loss: 3.392071E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.568 | TFLOPs: 32.05 |
7:  iteration    34800/  173500 | consumed samples:      8908800 | consumed tokens:  18245222400 | elapsed time per iteration (s): 0.37 | learning rate: 1.840E-04 | global batch size:   256 | lm loss: 3.390747E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.303 | TFLOPs: 32.22 |
7:  iteration    34900/  173500 | consumed samples:      8934400 | consumed tokens:  18297651200 | elapsed time per iteration (s): 0.37 | learning rate: 1.839E-04 | global batch size:   256 | lm loss: 3.386750E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.452 | TFLOPs: 32.13 |
7:  iteration    35000/  173500 | consumed samples:      8960000 | consumed tokens:  18350080000 | elapsed time per iteration (s): 0.37 | learning rate: 1.838E-04 | global batch size:   256 | lm loss: 3.389819E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.853 | TFLOPs: 32.39 |
7:  iteration    35100/  173500 | consumed samples:      8985600 | consumed tokens:  18402508800 | elapsed time per iteration (s): 0.37 | learning rate: 1.838E-04 | global batch size:   256 | lm loss: 3.388393E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.166 | TFLOPs: 32.35 |
7:  iteration    35200/  173500 | consumed samples:      9011200 | consumed tokens:  18454937600 | elapsed time per iteration (s): 0.37 | learning rate: 1.837E-04 | global batch size:   256 | lm loss: 3.385497E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.728 | TFLOPs: 32.38 |
7:  iteration    35300/  173500 | consumed samples:      9036800 | consumed tokens:  18507366400 | elapsed time per iteration (s): 0.37 | learning rate: 1.836E-04 | global batch size:   256 | lm loss: 3.389112E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.461 | TFLOPs: 32.23 |
7:  iteration    35400/  173500 | consumed samples:      9062400 | consumed tokens:  18559795200 | elapsed time per iteration (s): 0.37 | learning rate: 1.835E-04 | global batch size:   256 | lm loss: 3.387772E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.982 | TFLOPs: 32.39 |
7:  iteration    35500/  173500 | consumed samples:      9088000 | consumed tokens:  18612224000 | elapsed time per iteration (s): 0.37 | learning rate: 1.834E-04 | global batch size:   256 | lm loss: 3.388598E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.605 | TFLOPs: 32.37 |
7:  iteration    35600/  173500 | consumed samples:      9113600 | consumed tokens:  18664652800 | elapsed time per iteration (s): 0.37 | learning rate: 1.833E-04 | global batch size:   256 | lm loss: 3.382847E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.807 | TFLOPs: 32.38 |
7:  iteration    35700/  173500 | consumed samples:      9139200 | consumed tokens:  18717081600 | elapsed time per iteration (s): 0.37 | learning rate: 1.832E-04 | global batch size:   256 | lm loss: 3.382460E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.355 | TFLOPs: 32.41 |
7:  iteration    35800/  173500 | consumed samples:      9164800 | consumed tokens:  18769510400 | elapsed time per iteration (s): 0.37 | learning rate: 1.831E-04 | global batch size:   256 | lm loss: 3.383988E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.411 | TFLOPs: 32.23 |
7:  iteration    35900/  173500 | consumed samples:      9190400 | consumed tokens:  18821939200 | elapsed time per iteration (s): 0.37 | learning rate: 1.830E-04 | global batch size:   256 | lm loss: 3.386232E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.091 | TFLOPs: 32.35 |
0: [2023-03-17 03:07:10,523] [INFO] [logging.py:68:log_dist] [Rank 0] step=36000, skipped=0, lr=[0.00018289669072542715, 0.00018289669072542715, 0.00018289669072542715], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    36000/  173500 | consumed samples:      9216000 | consumed tokens:  18874368000 | elapsed time per iteration (s): 0.37 | learning rate: 1.829E-04 | global batch size:   256 | lm loss: 3.379866E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.432 | TFLOPs: 32.41 |
0: steps: 36000 loss: 3.3767 iter time (s): 0.369 samples/sec: 694.480
7:  iteration    36100/  173500 | consumed samples:      9241600 | consumed tokens:  18926796800 | elapsed time per iteration (s): 0.37 | learning rate: 1.828E-04 | global batch size:   256 | lm loss: 3.382685E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.740 | TFLOPs: 32.38 |
7:  iteration    36200/  173500 | consumed samples:      9267200 | consumed tokens:  18979225600 | elapsed time per iteration (s): 0.37 | learning rate: 1.827E-04 | global batch size:   256 | lm loss: 3.384482E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.111 | TFLOPs: 32.40 |
7:  iteration    36300/  173500 | consumed samples:      9292800 | consumed tokens:  19031654400 | elapsed time per iteration (s): 0.37 | learning rate: 1.826E-04 | global batch size:   256 | lm loss: 3.383112E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.535 | TFLOPs: 32.37 |
7:  iteration    36400/  173500 | consumed samples:      9318400 | consumed tokens:  19084083200 | elapsed time per iteration (s): 0.37 | learning rate: 1.825E-04 | global batch size:   256 | lm loss: 3.383809E+00 | grad norm: 0.260 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.217 | TFLOPs: 32.40 |
7:  iteration    36500/  173500 | consumed samples:      9344000 | consumed tokens:  19136512000 | elapsed time per iteration (s): 0.37 | learning rate: 1.824E-04 | global batch size:   256 | lm loss: 3.381419E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.423 | TFLOPs: 32.37 |
7:  iteration    36600/  173500 | consumed samples:      9369600 | consumed tokens:  19188940800 | elapsed time per iteration (s): 0.37 | learning rate: 1.823E-04 | global batch size:   256 | lm loss: 3.381317E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.161 | TFLOPs: 32.31 |
7:  iteration    36700/  173500 | consumed samples:      9395200 | consumed tokens:  19241369600 | elapsed time per iteration (s): 0.37 | learning rate: 1.822E-04 | global batch size:   256 | lm loss: 3.383944E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.939 | TFLOPs: 32.39 |
7:  iteration    36800/  173500 | consumed samples:      9420800 | consumed tokens:  19293798400 | elapsed time per iteration (s): 0.37 | learning rate: 1.821E-04 | global batch size:   256 | lm loss: 3.381586E+00 | grad norm: 0.246 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.037 | TFLOPs: 32.40 |
7:  iteration    36900/  173500 | consumed samples:      9446400 | consumed tokens:  19346227200 | elapsed time per iteration (s): 0.37 | learning rate: 1.820E-04 | global batch size:   256 | lm loss: 3.381295E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.605 | TFLOPs: 32.37 |
7:  iteration    37000/  173500 | consumed samples:      9472000 | consumed tokens:  19398656000 | elapsed time per iteration (s): 0.37 | learning rate: 1.819E-04 | global batch size:   256 | lm loss: 3.384636E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.689 | TFLOPs: 32.38 |
7:  iteration    37100/  173500 | consumed samples:      9497600 | consumed tokens:  19451084800 | elapsed time per iteration (s): 0.37 | learning rate: 1.818E-04 | global batch size:   256 | lm loss: 3.383159E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.561 | TFLOPs: 32.42 |
7:  iteration    37200/  173500 | consumed samples:      9523200 | consumed tokens:  19503513600 | elapsed time per iteration (s): 0.37 | learning rate: 1.817E-04 | global batch size:   256 | lm loss: 3.387179E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.870 | TFLOPs: 32.43 |
7:  iteration    37300/  173500 | consumed samples:      9548800 | consumed tokens:  19555942400 | elapsed time per iteration (s): 0.37 | learning rate: 1.816E-04 | global batch size:   256 | lm loss: 3.381431E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.128 | TFLOPs: 32.40 |
7:  iteration    37400/  173500 | consumed samples:      9574400 | consumed tokens:  19608371200 | elapsed time per iteration (s): 0.37 | learning rate: 1.815E-04 | global batch size:   256 | lm loss: 3.376826E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.518 | TFLOPs: 32.42 |
7:  iteration    37500/  173500 | consumed samples:      9600000 | consumed tokens:  19660800000 | elapsed time per iteration (s): 0.37 | learning rate: 1.814E-04 | global batch size:   256 | lm loss: 3.374964E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.351 | TFLOPs: 32.41 |
7:  iteration    37600/  173500 | consumed samples:      9625600 | consumed tokens:  19713228800 | elapsed time per iteration (s): 0.37 | learning rate: 1.813E-04 | global batch size:   256 | lm loss: 3.379276E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.553 | TFLOPs: 32.42 |
7:  iteration    37700/  173500 | consumed samples:      9651200 | consumed tokens:  19765657600 | elapsed time per iteration (s): 0.37 | learning rate: 1.812E-04 | global batch size:   256 | lm loss: 3.378383E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.430 | TFLOPs: 32.41 |
7:  iteration    37800/  173500 | consumed samples:      9676800 | consumed tokens:  19818086400 | elapsed time per iteration (s): 0.37 | learning rate: 1.811E-04 | global batch size:   256 | lm loss: 3.373248E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.477 | TFLOPs: 32.37 |
7:  iteration    37900/  173500 | consumed samples:      9702400 | consumed tokens:  19870515200 | elapsed time per iteration (s): 0.37 | learning rate: 1.810E-04 | global batch size:   256 | lm loss: 3.383512E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.015 | TFLOPs: 32.39 |
0: [2023-03-17 03:19:28,334] [INFO] [logging.py:68:log_dist] [Rank 0] step=38000, skipped=0, lr=[0.00018091754328052937, 0.00018091754328052937, 0.00018091754328052937], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    38000/  173500 | consumed samples:      9728000 | consumed tokens:  19922944000 | elapsed time per iteration (s): 0.37 | learning rate: 1.809E-04 | global batch size:   256 | lm loss: 3.378315E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.521 | TFLOPs: 32.37 |
0: steps: 38000 loss: 3.3848 iter time (s): 0.367 samples/sec: 697.436
7:  iteration    38100/  173500 | consumed samples:      9753600 | consumed tokens:  19975372800 | elapsed time per iteration (s): 0.37 | learning rate: 1.808E-04 | global batch size:   256 | lm loss: 3.378771E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.165 | TFLOPs: 32.35 |
7:  iteration    38200/  173500 | consumed samples:      9779200 | consumed tokens:  20027801600 | elapsed time per iteration (s): 0.37 | learning rate: 1.807E-04 | global batch size:   256 | lm loss: 3.376493E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.136 | TFLOPs: 32.40 |
7:  iteration    38300/  173500 | consumed samples:      9804800 | consumed tokens:  20080230400 | elapsed time per iteration (s): 0.37 | learning rate: 1.806E-04 | global batch size:   256 | lm loss: 3.372935E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.889 | TFLOPs: 32.39 |
7:  iteration    38400/  173500 | consumed samples:      9830400 | consumed tokens:  20132659200 | elapsed time per iteration (s): 0.37 | learning rate: 1.805E-04 | global batch size:   256 | lm loss: 3.376690E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.097 | TFLOPs: 32.30 |
7:  iteration    38500/  173500 | consumed samples:      9856000 | consumed tokens:  20185088000 | elapsed time per iteration (s): 0.37 | learning rate: 1.804E-04 | global batch size:   256 | lm loss: 3.377371E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.351 | TFLOPs: 32.41 |
7:  iteration    38600/  173500 | consumed samples:      9881600 | consumed tokens:  20237516800 | elapsed time per iteration (s): 0.37 | learning rate: 1.803E-04 | global batch size:   256 | lm loss: 3.375244E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.735 | TFLOPs: 32.38 |
7:  iteration    38700/  173500 | consumed samples:      9907200 | consumed tokens:  20289945600 | elapsed time per iteration (s): 0.37 | learning rate: 1.802E-04 | global batch size:   256 | lm loss: 3.377059E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.831 | TFLOPs: 32.34 |
7:  iteration    38800/  173500 | consumed samples:      9932800 | consumed tokens:  20342374400 | elapsed time per iteration (s): 0.37 | learning rate: 1.801E-04 | global batch size:   256 | lm loss: 3.376385E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.103 | TFLOPs: 32.35 |
7:  iteration    38900/  173500 | consumed samples:      9958400 | consumed tokens:  20394803200 | elapsed time per iteration (s): 0.37 | learning rate: 1.800E-04 | global batch size:   256 | lm loss: 3.372581E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.649 | TFLOPs: 32.33 |
7:  iteration    39000/  173500 | consumed samples:      9984000 | consumed tokens:  20447232000 | elapsed time per iteration (s): 0.37 | learning rate: 1.799E-04 | global batch size:   256 | lm loss: 3.371144E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.288 | TFLOPs: 32.27 |
7:  iteration    39100/  173500 | consumed samples:     10009600 | consumed tokens:  20499660800 | elapsed time per iteration (s): 0.37 | learning rate: 1.798E-04 | global batch size:   256 | lm loss: 3.372163E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.646 | TFLOPs: 32.19 |
7:  iteration    39200/  173500 | consumed samples:     10035200 | consumed tokens:  20552089600 | elapsed time per iteration (s): 0.37 | learning rate: 1.797E-04 | global batch size:   256 | lm loss: 3.376394E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.196 | TFLOPs: 32.31 |
7:  iteration    39300/  173500 | consumed samples:     10060800 | consumed tokens:  20604518400 | elapsed time per iteration (s): 0.37 | learning rate: 1.796E-04 | global batch size:   256 | lm loss: 3.373759E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.608 | TFLOPs: 32.33 |
7:  iteration    39400/  173500 | consumed samples:     10086400 | consumed tokens:  20656947200 | elapsed time per iteration (s): 0.37 | learning rate: 1.795E-04 | global batch size:   256 | lm loss: 3.371337E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.805 | TFLOPs: 32.34 |
7:  iteration    39500/  173500 | consumed samples:     10112000 | consumed tokens:  20709376000 | elapsed time per iteration (s): 0.37 | learning rate: 1.794E-04 | global batch size:   256 | lm loss: 3.372332E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.102 | TFLOPs: 32.35 |
7:  iteration    39600/  173500 | consumed samples:     10137600 | consumed tokens:  20761804800 | elapsed time per iteration (s): 0.37 | learning rate: 1.793E-04 | global batch size:   256 | lm loss: 3.369411E+00 | grad norm: 0.238 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.050 | TFLOPs: 32.35 |
7:  iteration    39700/  173500 | consumed samples:     10163200 | consumed tokens:  20814233600 | elapsed time per iteration (s): 0.37 | learning rate: 1.792E-04 | global batch size:   256 | lm loss: 3.374328E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.875 | TFLOPs: 32.34 |
7:  iteration    39800/  173500 | consumed samples:     10188800 | consumed tokens:  20866662400 | elapsed time per iteration (s): 0.37 | learning rate: 1.791E-04 | global batch size:   256 | lm loss: 3.370153E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.234 | TFLOPs: 32.36 |
7:  iteration    39900/  173500 | consumed samples:     10214400 | consumed tokens:  20919091200 | elapsed time per iteration (s): 0.37 | learning rate: 1.789E-04 | global batch size:   256 | lm loss: 3.370513E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.013 | TFLOPs: 32.35 |
0: [2023-03-17 03:31:47,309] [INFO] [logging.py:68:log_dist] [Rank 0] step=40000, skipped=0, lr=[0.0001788435118675357, 0.0001788435118675357, 0.0001788435118675357], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    40000/  173500 | consumed samples:     10240000 | consumed tokens:  20971520000 | elapsed time per iteration (s): 0.37 | learning rate: 1.788E-04 | global batch size:   256 | lm loss: 3.372030E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.319 | TFLOPs: 32.36 |
0: steps: 40000 loss: 3.3842 iter time (s): 0.367 samples/sec: 696.660
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 40000 | lm loss value: 3.427147E+00 | lm loss PPL: 3.078869E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   40000 to checkpoints_146m91b400m
0: [2023-03-17 03:31:47,434] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step40000 is begin to save!
0: [2023-03-17 03:31:47,439] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_01-model_00-model_states.pt...
0: [2023-03-17 03:31:47,535] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_01-model_00-model_states.pt.
0: [2023-03-17 03:31:47,536] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_03-model_00-model_states.pt...
0: [2023-03-17 03:31:47,551] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_03-model_00-model_states.pt.
0: [2023-03-17 03:31:47,551] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_04-model_00-model_states.pt...
0: [2023-03-17 03:31:47,566] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_04-model_00-model_states.pt.
0: [2023-03-17 03:31:47,566] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_05-model_00-model_states.pt...
0: [2023-03-17 03:31:47,581] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_05-model_00-model_states.pt.
0: [2023-03-17 03:31:47,581] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_06-model_00-model_states.pt...
0: [2023-03-17 03:31:47,596] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_06-model_00-model_states.pt.
0: [2023-03-17 03:31:47,596] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_07-model_00-model_states.pt...
0: [2023-03-17 03:31:47,611] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_07-model_00-model_states.pt.
0: [2023-03-17 03:31:47,612] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_08-model_00-model_states.pt...
0: [2023-03-17 03:31:47,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_08-model_00-model_states.pt.
0: [2023-03-17 03:31:47,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_09-model_00-model_states.pt...
0: [2023-03-17 03:31:47,641] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_09-model_00-model_states.pt.
0: [2023-03-17 03:31:47,642] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_10-model_00-model_states.pt...
0: [2023-03-17 03:31:47,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_10-model_00-model_states.pt.
0: [2023-03-17 03:31:47,657] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_11-model_00-model_states.pt...
0: [2023-03-17 03:31:47,671] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_11-model_00-model_states.pt.
0: [2023-03-17 03:31:47,672] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_12-model_00-model_states.pt...
0: [2023-03-17 03:31:47,686] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_12-model_00-model_states.pt.
0: [2023-03-17 03:31:47,687] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_13-model_00-model_states.pt...
0: [2023-03-17 03:31:47,701] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_13-model_00-model_states.pt.
0: [2023-03-17 03:31:47,702] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_14-model_00-model_states.pt...
0: [2023-03-17 03:31:47,716] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_14-model_00-model_states.pt.
0: [2023-03-17 03:31:47,717] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_15-model_00-model_states.pt...
0: [2023-03-17 03:31:47,731] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_15-model_00-model_states.pt.
0: [2023-03-17 03:31:47,732] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_16-model_00-model_states.pt...
0: [2023-03-17 03:31:47,746] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_16-model_00-model_states.pt.
0: [2023-03-17 03:31:47,747] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_17-model_00-model_states.pt...
0: [2023-03-17 03:31:47,761] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_17-model_00-model_states.pt.
0: [2023-03-17 03:31:47,762] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/layer_19-model_00-model_states.pt...
0: [2023-03-17 03:31:47,763] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/layer_19-model_00-model_states.pt.
0: [2023-03-17 03:31:47,763] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step40000/mp_rank_00_model_states.pt
0: [2023-03-17 03:31:47,763] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/mp_rank_00_model_states.pt...
0: [2023-03-17 03:31:47,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/mp_rank_00_model_states.pt.
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
7: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
5: [2023-03-17 03:31:47,785] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
0: [2023-03-17 03:31:47,818] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,818] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,818] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,819] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,821] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,821] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,821] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,822] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,822] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,822] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,824] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,824] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,825] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,825] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,825] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,825] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,825] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,825] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 03:31:47,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,826] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,826] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,828] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,828] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
4: [2023-03-17 03:31:47,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 03:31:47,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 03:31:47,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
2: [2023-03-17 03:31:47,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 03:31:47,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 03:31:47,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
3: [2023-03-17 03:31:47,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
6: [2023-03-17 03:31:47,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 03:31:47,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 03:31:47,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 03:31:47,854] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,854] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,854] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,854] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
1: [2023-03-17 03:31:47,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
7: [2023-03-17 03:31:47,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
5: [2023-03-17 03:31:47,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0: [2023-03-17 03:31:47,861] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 03:31:47,861] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now!
0:   successfully saved checkpoint at iteration   40000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 434.28
7:  iteration    40100/  173500 | consumed samples:     10265600 | consumed tokens:  21023948800 | elapsed time per iteration (s): 0.37 | learning rate: 1.787E-04 | global batch size:   256 | lm loss: 3.369131E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.771 | TFLOPs: 31.87 |
7:  iteration    40200/  173500 | consumed samples:     10291200 | consumed tokens:  21076377600 | elapsed time per iteration (s): 0.37 | learning rate: 1.786E-04 | global batch size:   256 | lm loss: 3.368906E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.056 | TFLOPs: 32.35 |
7:  iteration    40300/  173500 | consumed samples:     10316800 | consumed tokens:  21128806400 | elapsed time per iteration (s): 0.37 | learning rate: 1.785E-04 | global batch size:   256 | lm loss: 3.370228E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.519 | TFLOPs: 32.32 |
7:  iteration    40400/  173500 | consumed samples:     10342400 | consumed tokens:  21181235200 | elapsed time per iteration (s): 0.37 | learning rate: 1.784E-04 | global batch size:   256 | lm loss: 3.371949E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.437 | TFLOPs: 32.37 |
7:  iteration    40500/  173500 | consumed samples:     10368000 | consumed tokens:  21233664000 | elapsed time per iteration (s): 0.37 | learning rate: 1.783E-04 | global batch size:   256 | lm loss: 3.368385E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.596 | TFLOPs: 32.37 |
7:  iteration    40600/  173500 | consumed samples:     10393600 | consumed tokens:  21286092800 | elapsed time per iteration (s): 0.37 | learning rate: 1.782E-04 | global batch size:   256 | lm loss: 3.371410E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.932 | TFLOPs: 32.34 |
7:  iteration    40700/  173500 | consumed samples:     10419200 | consumed tokens:  21338521600 | elapsed time per iteration (s): 0.37 | learning rate: 1.781E-04 | global batch size:   256 | lm loss: 3.368692E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.346 | TFLOPs: 32.22 |
7:  iteration    40800/  173500 | consumed samples:     10444800 | consumed tokens:  21390950400 | elapsed time per iteration (s): 0.37 | learning rate: 1.780E-04 | global batch size:   256 | lm loss: 3.368367E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.837 | TFLOPs: 32.39 |
7:  iteration    40900/  173500 | consumed samples:     10470400 | consumed tokens:  21443379200 | elapsed time per iteration (s): 0.37 | learning rate: 1.779E-04 | global batch size:   256 | lm loss: 3.368872E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.114 | TFLOPs: 32.40 |
7:  iteration    41000/  173500 | consumed samples:     10496000 | consumed tokens:  21495808000 | elapsed time per iteration (s): 0.37 | learning rate: 1.778E-04 | global batch size:   256 | lm loss: 3.366469E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.414 | TFLOPs: 32.32 |
7:  iteration    41100/  173500 | consumed samples:     10521600 | consumed tokens:  21548236800 | elapsed time per iteration (s): 0.37 | learning rate: 1.777E-04 | global batch size:   256 | lm loss: 3.363399E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.274 | TFLOPs: 32.13 |
7:  iteration    41200/  173500 | consumed samples:     10547200 | consumed tokens:  21600665600 | elapsed time per iteration (s): 0.37 | learning rate: 1.776E-04 | global batch size:   256 | lm loss: 3.367148E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.864 | TFLOPs: 32.25 |
7:  iteration    41300/  173500 | consumed samples:     10572800 | consumed tokens:  21653094400 | elapsed time per iteration (s): 0.37 | learning rate: 1.774E-04 | global batch size:   256 | lm loss: 3.365396E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.904 | TFLOPs: 32.34 |
7:  iteration    41400/  173500 | consumed samples:     10598400 | consumed tokens:  21705523200 | elapsed time per iteration (s): 0.37 | learning rate: 1.773E-04 | global batch size:   256 | lm loss: 3.366451E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.056 | TFLOPs: 32.35 |
7:  iteration    41500/  173500 | consumed samples:     10624000 | consumed tokens:  21757952000 | elapsed time per iteration (s): 0.37 | learning rate: 1.772E-04 | global batch size:   256 | lm loss: 3.366326E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.097 | TFLOPs: 32.30 |
7:  iteration    41600/  173500 | consumed samples:     10649600 | consumed tokens:  21810380800 | elapsed time per iteration (s): 0.37 | learning rate: 1.771E-04 | global batch size:   256 | lm loss: 3.364752E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.836 | TFLOPs: 32.25 |
7:  iteration    41700/  173500 | consumed samples:     10675200 | consumed tokens:  21862809600 | elapsed time per iteration (s): 0.37 | learning rate: 1.770E-04 | global batch size:   256 | lm loss: 3.364167E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.250 | TFLOPs: 32.27 |
7:  iteration    41800/  173500 | consumed samples:     10700800 | consumed tokens:  21915238400 | elapsed time per iteration (s): 0.37 | learning rate: 1.769E-04 | global batch size:   256 | lm loss: 3.361969E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.469 | TFLOPs: 32.32 |
7:  iteration    41900/  173500 | consumed samples:     10726400 | consumed tokens:  21967667200 | elapsed time per iteration (s): 0.37 | learning rate: 1.768E-04 | global batch size:   256 | lm loss: 3.365338E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.933 | TFLOPs: 31.97 |
0: [2023-03-17 03:44:08,225] [INFO] [logging.py:68:log_dist] [Rank 0] step=42000, skipped=0, lr=[0.00017667737143212697, 0.00017667737143212697, 0.00017667737143212697], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    42000/  173500 | consumed samples:     10752000 | consumed tokens:  22020096000 | elapsed time per iteration (s): 0.37 | learning rate: 1.767E-04 | global batch size:   256 | lm loss: 3.370497E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.292 | TFLOPs: 31.99 |
0: steps: 42000 loss: 3.3481 iter time (s): 0.368 samples/sec: 695.135
7:  iteration    42100/  173500 | consumed samples:     10777600 | consumed tokens:  22072524800 | elapsed time per iteration (s): 0.38 | learning rate: 1.766E-04 | global batch size:   256 | lm loss: 3.364428E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.438 | TFLOPs: 31.85 |
7:  iteration    42200/  173500 | consumed samples:     10803200 | consumed tokens:  22124953600 | elapsed time per iteration (s): 0.37 | learning rate: 1.765E-04 | global batch size:   256 | lm loss: 3.362214E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.024 | TFLOPs: 32.07 |
7:  iteration    42300/  173500 | consumed samples:     10828800 | consumed tokens:  22177382400 | elapsed time per iteration (s): 0.37 | learning rate: 1.763E-04 | global batch size:   256 | lm loss: 3.364200E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.076 | TFLOPs: 32.02 |
7:  iteration    42400/  173500 | consumed samples:     10854400 | consumed tokens:  22229811200 | elapsed time per iteration (s): 0.37 | learning rate: 1.762E-04 | global batch size:   256 | lm loss: 3.359431E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.336 | TFLOPs: 32.13 |
7:  iteration    42500/  173500 | consumed samples:     10880000 | consumed tokens:  22282240000 | elapsed time per iteration (s): 0.37 | learning rate: 1.761E-04 | global batch size:   256 | lm loss: 3.359092E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.816 | TFLOPs: 32.24 |
7:  iteration    42600/  173500 | consumed samples:     10905600 | consumed tokens:  22334668800 | elapsed time per iteration (s): 0.37 | learning rate: 1.760E-04 | global batch size:   256 | lm loss: 3.364756E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.832 | TFLOPs: 32.11 |
7:  iteration    42700/  173500 | consumed samples:     10931200 | consumed tokens:  22387097600 | elapsed time per iteration (s): 0.37 | learning rate: 1.759E-04 | global batch size:   256 | lm loss: 3.357336E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.319 | TFLOPs: 32.17 |
7:  iteration    42800/  173500 | consumed samples:     10956800 | consumed tokens:  22439526400 | elapsed time per iteration (s): 0.37 | learning rate: 1.758E-04 | global batch size:   256 | lm loss: 3.364207E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.287 | TFLOPs: 32.03 |
7:  iteration    42900/  173500 | consumed samples:     10982400 | consumed tokens:  22491955200 | elapsed time per iteration (s): 0.37 | learning rate: 1.757E-04 | global batch size:   256 | lm loss: 3.360787E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.848 | TFLOPs: 32.06 |
7:  iteration    43000/  173500 | consumed samples:     11008000 | consumed tokens:  22544384000 | elapsed time per iteration (s): 0.37 | learning rate: 1.756E-04 | global batch size:   256 | lm loss: 3.359987E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.030 | TFLOPs: 32.07 |
7:  iteration    43100/  173500 | consumed samples:     11033600 | consumed tokens:  22596812800 | elapsed time per iteration (s): 0.38 | learning rate: 1.754E-04 | global batch size:   256 | lm loss: 3.359879E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 676.754 | TFLOPs: 31.59 |
7:  iteration    43200/  173500 | consumed samples:     11059200 | consumed tokens:  22649241600 | elapsed time per iteration (s): 0.38 | learning rate: 1.753E-04 | global batch size:   256 | lm loss: 3.361799E+00 | grad norm: 0.250 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.547 | TFLOPs: 31.72 |
7:  iteration    43300/  173500 | consumed samples:     11084800 | consumed tokens:  22701670400 | elapsed time per iteration (s): 0.38 | learning rate: 1.752E-04 | global batch size:   256 | lm loss: 3.358792E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.438 | TFLOPs: 31.85 |
7:  iteration    43400/  173500 | consumed samples:     11110400 | consumed tokens:  22754099200 | elapsed time per iteration (s): 0.38 | learning rate: 1.751E-04 | global batch size:   256 | lm loss: 3.361978E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.402 | TFLOPs: 31.81 |
7:  iteration    43500/  173500 | consumed samples:     11136000 | consumed tokens:  22806528000 | elapsed time per iteration (s): 0.38 | learning rate: 1.750E-04 | global batch size:   256 | lm loss: 3.362261E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.644 | TFLOPs: 31.68 |
7:  iteration    43600/  173500 | consumed samples:     11161600 | consumed tokens:  22858956800 | elapsed time per iteration (s): 0.38 | learning rate: 1.749E-04 | global batch size:   256 | lm loss: 3.357797E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.496 | TFLOPs: 31.76 |
7:  iteration    43700/  173500 | consumed samples:     11187200 | consumed tokens:  22911385600 | elapsed time per iteration (s): 0.38 | learning rate: 1.748E-04 | global batch size:   256 | lm loss: 3.354324E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 666.890 | TFLOPs: 31.13 |
7:  iteration    43800/  173500 | consumed samples:     11212800 | consumed tokens:  22963814400 | elapsed time per iteration (s): 0.38 | learning rate: 1.747E-04 | global batch size:   256 | lm loss: 3.356634E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.407 | TFLOPs: 31.67 |
7:  iteration    43900/  173500 | consumed samples:     11238400 | consumed tokens:  23016243200 | elapsed time per iteration (s): 0.38 | learning rate: 1.745E-04 | global batch size:   256 | lm loss: 3.356181E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.716 | TFLOPs: 31.68 |
0: [2023-03-17 03:56:38,148] [INFO] [logging.py:68:log_dist] [Rank 0] step=44000, skipped=0, lr=[0.00017442202015704406, 0.00017442202015704406, 0.00017442202015704406], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    44000/  173500 | consumed samples:     11264000 | consumed tokens:  23068672000 | elapsed time per iteration (s): 0.38 | learning rate: 1.744E-04 | global batch size:   256 | lm loss: 3.356203E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.277 | TFLOPs: 31.75 |
0: steps: 44000 loss: 3.3076 iter time (s): 0.373 samples/sec: 686.550
7:  iteration    44100/  173500 | consumed samples:     11289600 | consumed tokens:  23121100800 | elapsed time per iteration (s): 0.38 | learning rate: 1.743E-04 | global batch size:   256 | lm loss: 3.354622E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.888 | TFLOPs: 31.83 |
7:  iteration    44200/  173500 | consumed samples:     11315200 | consumed tokens:  23173529600 | elapsed time per iteration (s): 0.37 | learning rate: 1.742E-04 | global batch size:   256 | lm loss: 3.353467E+00 | grad norm: 0.191 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.755 | TFLOPs: 31.87 |
7:  iteration    44300/  173500 | consumed samples:     11340800 | consumed tokens:  23225958400 | elapsed time per iteration (s): 0.37 | learning rate: 1.741E-04 | global batch size:   256 | lm loss: 3.355652E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.276 | TFLOPs: 32.08 |
7:  iteration    44400/  173500 | consumed samples:     11366400 | consumed tokens:  23278387200 | elapsed time per iteration (s): 0.37 | learning rate: 1.740E-04 | global batch size:   256 | lm loss: 3.354942E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.104 | TFLOPs: 31.98 |
7:  iteration    44500/  173500 | consumed samples:     11392000 | consumed tokens:  23330816000 | elapsed time per iteration (s): 0.37 | learning rate: 1.738E-04 | global batch size:   256 | lm loss: 3.356569E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.161 | TFLOPs: 31.89 |
7:  iteration    44600/  173500 | consumed samples:     11417600 | consumed tokens:  23383244800 | elapsed time per iteration (s): 0.37 | learning rate: 1.737E-04 | global batch size:   256 | lm loss: 3.351849E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.198 | TFLOPs: 32.12 |
7:  iteration    44700/  173500 | consumed samples:     11443200 | consumed tokens:  23435673600 | elapsed time per iteration (s): 0.37 | learning rate: 1.736E-04 | global batch size:   256 | lm loss: 3.356621E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.552 | TFLOPs: 32.23 |
7:  iteration    44800/  173500 | consumed samples:     11468800 | consumed tokens:  23488102400 | elapsed time per iteration (s): 0.37 | learning rate: 1.735E-04 | global batch size:   256 | lm loss: 3.354205E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.895 | TFLOPs: 32.02 |
7:  iteration    44900/  173500 | consumed samples:     11494400 | consumed tokens:  23540531200 | elapsed time per iteration (s): 0.37 | learning rate: 1.734E-04 | global batch size:   256 | lm loss: 3.352970E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.617 | TFLOPs: 32.19 |
7:  iteration    45000/  173500 | consumed samples:     11520000 | consumed tokens:  23592960000 | elapsed time per iteration (s): 0.37 | learning rate: 1.733E-04 | global batch size:   256 | lm loss: 3.353456E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.746 | TFLOPs: 32.24 |
7:  iteration    45100/  173500 | consumed samples:     11545600 | consumed tokens:  23645388800 | elapsed time per iteration (s): 0.37 | learning rate: 1.731E-04 | global batch size:   256 | lm loss: 3.354372E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.759 | TFLOPs: 32.20 |
7:  iteration    45200/  173500 | consumed samples:     11571200 | consumed tokens:  23697817600 | elapsed time per iteration (s): 0.37 | learning rate: 1.730E-04 | global batch size:   256 | lm loss: 3.355781E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.650 | TFLOPs: 32.14 |
7:  iteration    45300/  173500 | consumed samples:     11596800 | consumed tokens:  23750246400 | elapsed time per iteration (s): 0.37 | learning rate: 1.729E-04 | global batch size:   256 | lm loss: 3.356636E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.463 | TFLOPs: 32.23 |
7:  iteration    45400/  173500 | consumed samples:     11622400 | consumed tokens:  23802675200 | elapsed time per iteration (s): 0.37 | learning rate: 1.728E-04 | global batch size:   256 | lm loss: 3.354018E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.357 | TFLOPs: 32.04 |
7:  iteration    45500/  173500 | consumed samples:     11648000 | consumed tokens:  23855104000 | elapsed time per iteration (s): 0.37 | learning rate: 1.727E-04 | global batch size:   256 | lm loss: 3.351548E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.238 | TFLOPs: 32.22 |
7:  iteration    45600/  173500 | consumed samples:     11673600 | consumed tokens:  23907532800 | elapsed time per iteration (s): 0.37 | learning rate: 1.726E-04 | global batch size:   256 | lm loss: 3.350341E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.119 | TFLOPs: 32.17 |
7:  iteration    45700/  173500 | consumed samples:     11699200 | consumed tokens:  23959961600 | elapsed time per iteration (s): 0.37 | learning rate: 1.724E-04 | global batch size:   256 | lm loss: 3.351876E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.920 | TFLOPs: 32.34 |
7:  iteration    45800/  173500 | consumed samples:     11724800 | consumed tokens:  24012390400 | elapsed time per iteration (s): 0.37 | learning rate: 1.723E-04 | global batch size:   256 | lm loss: 3.351292E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.009 | TFLOPs: 32.30 |
7:  iteration    45900/  173500 | consumed samples:     11750400 | consumed tokens:  24064819200 | elapsed time per iteration (s): 0.37 | learning rate: 1.722E-04 | global batch size:   256 | lm loss: 3.351722E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.489 | TFLOPs: 32.32 |
0: [2023-03-17 04:09:01,767] [INFO] [logging.py:68:log_dist] [Rank 0] step=46000, skipped=0, lr=[0.00017208047558447097, 0.00017208047558447097, 0.00017208047558447097], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    46000/  173500 | consumed samples:     11776000 | consumed tokens:  24117248000 | elapsed time per iteration (s): 0.37 | learning rate: 1.721E-04 | global batch size:   256 | lm loss: 3.348051E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.715 | TFLOPs: 32.38 |
0: steps: 46000 loss: 3.3195 iter time (s): 0.370 samples/sec: 692.181
7:  iteration    46100/  173500 | consumed samples:     11801600 | consumed tokens:  24169676800 | elapsed time per iteration (s): 0.37 | learning rate: 1.720E-04 | global batch size:   256 | lm loss: 3.351471E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.326 | TFLOPs: 32.36 |
7:  iteration    46200/  173500 | consumed samples:     11827200 | consumed tokens:  24222105600 | elapsed time per iteration (s): 0.37 | learning rate: 1.718E-04 | global batch size:   256 | lm loss: 3.352455E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.452 | TFLOPs: 32.37 |
7:  iteration    46300/  173500 | consumed samples:     11852800 | consumed tokens:  24274534400 | elapsed time per iteration (s): 0.37 | learning rate: 1.717E-04 | global batch size:   256 | lm loss: 3.351888E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.034 | TFLOPs: 32.30 |
7:  iteration    46400/  173500 | consumed samples:     11878400 | consumed tokens:  24326963200 | elapsed time per iteration (s): 0.37 | learning rate: 1.716E-04 | global batch size:   256 | lm loss: 3.346888E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.525 | TFLOPs: 32.37 |
7:  iteration    46500/  173500 | consumed samples:     11904000 | consumed tokens:  24379392000 | elapsed time per iteration (s): 0.37 | learning rate: 1.715E-04 | global batch size:   256 | lm loss: 3.353758E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.015 | TFLOPs: 32.35 |
7:  iteration    46600/  173500 | consumed samples:     11929600 | consumed tokens:  24431820800 | elapsed time per iteration (s): 0.37 | learning rate: 1.714E-04 | global batch size:   256 | lm loss: 3.350063E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.830 | TFLOPs: 32.34 |
7:  iteration    46700/  173500 | consumed samples:     11955200 | consumed tokens:  24484249600 | elapsed time per iteration (s): 0.37 | learning rate: 1.712E-04 | global batch size:   256 | lm loss: 3.351083E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.771 | TFLOPs: 32.38 |
7:  iteration    46800/  173500 | consumed samples:     11980800 | consumed tokens:  24536678400 | elapsed time per iteration (s): 0.37 | learning rate: 1.711E-04 | global batch size:   256 | lm loss: 3.348693E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.712 | TFLOPs: 32.38 |
7:  iteration    46900/  173500 | consumed samples:     12006400 | consumed tokens:  24589107200 | elapsed time per iteration (s): 0.37 | learning rate: 1.710E-04 | global batch size:   256 | lm loss: 3.352725E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.858 | TFLOPs: 32.06 |
7:  iteration    47000/  173500 | consumed samples:     12032000 | consumed tokens:  24641536000 | elapsed time per iteration (s): 0.37 | learning rate: 1.709E-04 | global batch size:   256 | lm loss: 3.348880E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.076 | TFLOPs: 32.26 |
7:  iteration    47100/  173500 | consumed samples:     12057600 | consumed tokens:  24693964800 | elapsed time per iteration (s): 0.37 | learning rate: 1.708E-04 | global batch size:   256 | lm loss: 3.349565E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.876 | TFLOPs: 32.39 |
7:  iteration    47200/  173500 | consumed samples:     12083200 | consumed tokens:  24746393600 | elapsed time per iteration (s): 0.37 | learning rate: 1.706E-04 | global batch size:   256 | lm loss: 3.344828E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.422 | TFLOPs: 32.37 |
7:  iteration    47300/  173500 | consumed samples:     12108800 | consumed tokens:  24798822400 | elapsed time per iteration (s): 0.37 | learning rate: 1.705E-04 | global batch size:   256 | lm loss: 3.348427E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.352 | TFLOPs: 32.36 |
7:  iteration    47400/  173500 | consumed samples:     12134400 | consumed tokens:  24851251200 | elapsed time per iteration (s): 0.37 | learning rate: 1.704E-04 | global batch size:   256 | lm loss: 3.349526E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.400 | TFLOPs: 32.37 |
7:  iteration    47500/  173500 | consumed samples:     12160000 | consumed tokens:  24903680000 | elapsed time per iteration (s): 0.37 | learning rate: 1.703E-04 | global batch size:   256 | lm loss: 3.346141E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.724 | TFLOPs: 32.38 |
7:  iteration    47600/  173500 | consumed samples:     12185600 | consumed tokens:  24956108800 | elapsed time per iteration (s): 0.37 | learning rate: 1.701E-04 | global batch size:   256 | lm loss: 3.347261E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.739 | TFLOPs: 32.38 |
7:  iteration    47700/  173500 | consumed samples:     12211200 | consumed tokens:  25008537600 | elapsed time per iteration (s): 0.37 | learning rate: 1.700E-04 | global batch size:   256 | lm loss: 3.346654E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.644 | TFLOPs: 32.38 |
7:  iteration    47800/  173500 | consumed samples:     12236800 | consumed tokens:  25060966400 | elapsed time per iteration (s): 0.37 | learning rate: 1.699E-04 | global batch size:   256 | lm loss: 3.342170E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.606 | TFLOPs: 32.37 |
7:  iteration    47900/  173500 | consumed samples:     12262400 | consumed tokens:  25113395200 | elapsed time per iteration (s): 0.37 | learning rate: 1.698E-04 | global batch size:   256 | lm loss: 3.346464E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.805 | TFLOPs: 32.38 |
0: [2023-03-17 04:21:20,602] [INFO] [logging.py:68:log_dist] [Rank 0] step=48000, skipped=0, lr=[0.00016965587057872074, 0.00016965587057872074, 0.00016965587057872074], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    48000/  173500 | consumed samples:     12288000 | consumed tokens:  25165824000 | elapsed time per iteration (s): 0.37 | learning rate: 1.697E-04 | global batch size:   256 | lm loss: 3.343630E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.572 | TFLOPs: 32.37 |
0: steps: 48000 loss: 3.3092 iter time (s): 0.368 samples/sec: 696.554
7:  iteration    48100/  173500 | consumed samples:     12313600 | consumed tokens:  25218252800 | elapsed time per iteration (s): 0.37 | learning rate: 1.695E-04 | global batch size:   256 | lm loss: 3.346831E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.212 | TFLOPs: 32.36 |
7:  iteration    48200/  173500 | consumed samples:     12339200 | consumed tokens:  25270681600 | elapsed time per iteration (s): 0.37 | learning rate: 1.694E-04 | global batch size:   256 | lm loss: 3.349657E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.198 | TFLOPs: 32.36 |
7:  iteration    48300/  173500 | consumed samples:     12364800 | consumed tokens:  25323110400 | elapsed time per iteration (s): 0.37 | learning rate: 1.693E-04 | global batch size:   256 | lm loss: 3.342500E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.231 | TFLOPs: 32.36 |
7:  iteration    48400/  173500 | consumed samples:     12390400 | consumed tokens:  25375539200 | elapsed time per iteration (s): 0.37 | learning rate: 1.692E-04 | global batch size:   256 | lm loss: 3.340730E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.493 | TFLOPs: 32.37 |
7:  iteration    48500/  173500 | consumed samples:     12416000 | consumed tokens:  25427968000 | elapsed time per iteration (s): 0.37 | learning rate: 1.690E-04 | global batch size:   256 | lm loss: 3.342486E+00 | grad norm: 0.251 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.495 | TFLOPs: 32.32 |
7:  iteration    48600/  173500 | consumed samples:     12441600 | consumed tokens:  25480396800 | elapsed time per iteration (s): 0.37 | learning rate: 1.689E-04 | global batch size:   256 | lm loss: 3.342491E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.596 | TFLOPs: 32.23 |
7:  iteration    48700/  173500 | consumed samples:     12467200 | consumed tokens:  25532825600 | elapsed time per iteration (s): 0.37 | learning rate: 1.688E-04 | global batch size:   256 | lm loss: 3.340196E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.220 | TFLOPs: 32.17 |
7:  iteration    48800/  173500 | consumed samples:     12492800 | consumed tokens:  25585254400 | elapsed time per iteration (s): 0.37 | learning rate: 1.687E-04 | global batch size:   256 | lm loss: 3.340405E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.960 | TFLOPs: 32.34 |
7:  iteration    48900/  173500 | consumed samples:     12518400 | consumed tokens:  25637683200 | elapsed time per iteration (s): 0.37 | learning rate: 1.685E-04 | global batch size:   256 | lm loss: 3.340890E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.146 | TFLOPs: 32.35 |
7:  iteration    49000/  173500 | consumed samples:     12544000 | consumed tokens:  25690112000 | elapsed time per iteration (s): 0.37 | learning rate: 1.684E-04 | global batch size:   256 | lm loss: 3.345238E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.038 | TFLOPs: 32.35 |
7:  iteration    49100/  173500 | consumed samples:     12569600 | consumed tokens:  25742540800 | elapsed time per iteration (s): 0.37 | learning rate: 1.683E-04 | global batch size:   256 | lm loss: 3.342634E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.552 | TFLOPs: 32.37 |
7:  iteration    49200/  173500 | consumed samples:     12595200 | consumed tokens:  25794969600 | elapsed time per iteration (s): 0.37 | learning rate: 1.682E-04 | global batch size:   256 | lm loss: 3.342739E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.480 | TFLOPs: 32.32 |
7:  iteration    49300/  173500 | consumed samples:     12620800 | consumed tokens:  25847398400 | elapsed time per iteration (s): 0.37 | learning rate: 1.680E-04 | global batch size:   256 | lm loss: 3.341374E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.268 | TFLOPs: 32.36 |
7:  iteration    49400/  173500 | consumed samples:     12646400 | consumed tokens:  25899827200 | elapsed time per iteration (s): 0.37 | learning rate: 1.679E-04 | global batch size:   256 | lm loss: 3.341710E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.740 | TFLOPs: 32.29 |
7:  iteration    49500/  173500 | consumed samples:     12672000 | consumed tokens:  25952256000 | elapsed time per iteration (s): 0.37 | learning rate: 1.678E-04 | global batch size:   256 | lm loss: 3.338491E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.156 | TFLOPs: 32.35 |
7:  iteration    49600/  173500 | consumed samples:     12697600 | consumed tokens:  26004684800 | elapsed time per iteration (s): 0.37 | learning rate: 1.677E-04 | global batch size:   256 | lm loss: 3.343585E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.607 | TFLOPs: 32.23 |
7:  iteration    49700/  173500 | consumed samples:     12723200 | consumed tokens:  26057113600 | elapsed time per iteration (s): 0.37 | learning rate: 1.675E-04 | global batch size:   256 | lm loss: 3.337025E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.120 | TFLOPs: 32.26 |
7:  iteration    49800/  173500 | consumed samples:     12748800 | consumed tokens:  26109542400 | elapsed time per iteration (s): 0.37 | learning rate: 1.674E-04 | global batch size:   256 | lm loss: 3.340688E+00 | grad norm: 0.250 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.192 | TFLOPs: 32.31 |
7:  iteration    49900/  173500 | consumed samples:     12774400 | consumed tokens:  26161971200 | elapsed time per iteration (s): 0.37 | learning rate: 1.673E-04 | global batch size:   256 | lm loss: 3.336629E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.981 | TFLOPs: 32.30 |
0: [2023-03-17 04:33:40,195] [INFO] [logging.py:68:log_dist] [Rank 0] step=50000, skipped=0, lr=[0.00016715144913462704, 0.00016715144913462704, 0.00016715144913462704], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    50000/  173500 | consumed samples:     12800000 | consumed tokens:  26214400000 | elapsed time per iteration (s): 0.37 | learning rate: 1.672E-04 | global batch size:   256 | lm loss: 3.341769E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.779 | TFLOPs: 32.24 |
0: steps: 50000 loss: 3.3590 iter time (s): 0.368 samples/sec: 695.799
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 50000 | lm loss value: 3.402776E+00 | lm loss PPL: 3.004741E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   50000 to checkpoints_146m91b400m
0: [2023-03-17 04:33:40,360] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step50000 is begin to save!
0: [2023-03-17 04:33:40,365] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_01-model_00-model_states.pt...
0: [2023-03-17 04:33:40,475] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_01-model_00-model_states.pt.
0: [2023-03-17 04:33:40,475] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_03-model_00-model_states.pt...
0: [2023-03-17 04:33:40,491] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_03-model_00-model_states.pt.
0: [2023-03-17 04:33:40,491] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_04-model_00-model_states.pt...
0: [2023-03-17 04:33:40,506] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_04-model_00-model_states.pt.
0: [2023-03-17 04:33:40,506] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_05-model_00-model_states.pt...
0: [2023-03-17 04:33:40,521] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_05-model_00-model_states.pt.
0: [2023-03-17 04:33:40,521] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_06-model_00-model_states.pt...
0: [2023-03-17 04:33:40,536] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_06-model_00-model_states.pt.
0: [2023-03-17 04:33:40,537] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_07-model_00-model_states.pt...
0: [2023-03-17 04:33:40,552] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_07-model_00-model_states.pt.
0: [2023-03-17 04:33:40,552] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_08-model_00-model_states.pt...
0: [2023-03-17 04:33:40,567] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_08-model_00-model_states.pt.
0: [2023-03-17 04:33:40,567] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_09-model_00-model_states.pt...
0: [2023-03-17 04:33:40,582] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_09-model_00-model_states.pt.
0: [2023-03-17 04:33:40,582] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_10-model_00-model_states.pt...
0: [2023-03-17 04:33:40,597] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_10-model_00-model_states.pt.
0: [2023-03-17 04:33:40,598] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_11-model_00-model_states.pt...
0: [2023-03-17 04:33:40,613] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_11-model_00-model_states.pt.
0: [2023-03-17 04:33:40,613] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_12-model_00-model_states.pt...
0: [2023-03-17 04:33:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_12-model_00-model_states.pt.
0: [2023-03-17 04:33:40,628] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_13-model_00-model_states.pt...
0: [2023-03-17 04:33:40,643] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_13-model_00-model_states.pt.
0: [2023-03-17 04:33:40,643] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_14-model_00-model_states.pt...
0: [2023-03-17 04:33:40,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_14-model_00-model_states.pt.
0: [2023-03-17 04:33:40,659] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_15-model_00-model_states.pt...
0: [2023-03-17 04:33:40,674] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_15-model_00-model_states.pt.
0: [2023-03-17 04:33:40,674] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_16-model_00-model_states.pt...
0: [2023-03-17 04:33:40,689] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_16-model_00-model_states.pt.
0: [2023-03-17 04:33:40,689] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_17-model_00-model_states.pt...
0: [2023-03-17 04:33:40,704] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_17-model_00-model_states.pt.
0: [2023-03-17 04:33:40,704] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/layer_19-model_00-model_states.pt...
0: [2023-03-17 04:33:40,706] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/layer_19-model_00-model_states.pt.
0: [2023-03-17 04:33:40,706] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step50000/mp_rank_00_model_states.pt
0: [2023-03-17 04:33:40,706] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/mp_rank_00_model_states.pt...
0: [2023-03-17 04:33:40,711] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/mp_rank_00_model_states.pt.
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
2: [2023-03-17 04:33:40,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,775] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,775] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,775] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,781] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 04:33:40,781] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,781] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,781] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,786] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,787] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
1: [2023-03-17 04:33:40,788] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 04:33:40,788] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 04:33:40,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
2: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 04:33:40,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0: [2023-03-17 04:33:40,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 04:33:40,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,796] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
7: [2023-03-17 04:33:40,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 04:33:40,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 04:33:40,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 04:33:40,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 04:33:40,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
6: [2023-03-17 04:33:40,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
4: [2023-03-17 04:33:40,802] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
5: [2023-03-17 04:33:40,820] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
3: [2023-03-17 04:33:40,823] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now!
0:   successfully saved checkpoint at iteration   50000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 509.04
7:  iteration    50100/  173500 | consumed samples:     12825600 | consumed tokens:  26266828800 | elapsed time per iteration (s): 0.38 | learning rate: 1.670E-04 | global batch size:   256 | lm loss: 3.335900E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.305 | TFLOPs: 31.75 |
7:  iteration    50200/  173500 | consumed samples:     12851200 | consumed tokens:  26319257600 | elapsed time per iteration (s): 0.37 | learning rate: 1.669E-04 | global batch size:   256 | lm loss: 3.341022E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.231 | TFLOPs: 32.36 |
7:  iteration    50300/  173500 | consumed samples:     12876800 | consumed tokens:  26371686400 | elapsed time per iteration (s): 0.37 | learning rate: 1.668E-04 | global batch size:   256 | lm loss: 3.339592E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.580 | TFLOPs: 32.37 |
7:  iteration    50400/  173500 | consumed samples:     12902400 | consumed tokens:  26424115200 | elapsed time per iteration (s): 0.37 | learning rate: 1.666E-04 | global batch size:   256 | lm loss: 3.339116E+00 | grad norm: 0.250 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.484 | TFLOPs: 32.37 |
7:  iteration    50500/  173500 | consumed samples:     12928000 | consumed tokens:  26476544000 | elapsed time per iteration (s): 0.37 | learning rate: 1.665E-04 | global batch size:   256 | lm loss: 3.337925E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.293 | TFLOPs: 32.36 |
7:  iteration    50600/  173500 | consumed samples:     12953600 | consumed tokens:  26528972800 | elapsed time per iteration (s): 0.37 | learning rate: 1.664E-04 | global batch size:   256 | lm loss: 3.335914E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.497 | TFLOPs: 32.18 |
7:  iteration    50700/  173500 | consumed samples:     12979200 | consumed tokens:  26581401600 | elapsed time per iteration (s): 0.37 | learning rate: 1.663E-04 | global batch size:   256 | lm loss: 3.339157E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.940 | TFLOPs: 32.11 |
7:  iteration    50800/  173500 | consumed samples:     13004800 | consumed tokens:  26633830400 | elapsed time per iteration (s): 0.37 | learning rate: 1.661E-04 | global batch size:   256 | lm loss: 3.335979E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.085 | TFLOPs: 32.40 |
7:  iteration    50900/  173500 | consumed samples:     13030400 | consumed tokens:  26686259200 | elapsed time per iteration (s): 0.37 | learning rate: 1.660E-04 | global batch size:   256 | lm loss: 3.337099E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.451 | TFLOPs: 32.41 |
7:  iteration    51000/  173500 | consumed samples:     13056000 | consumed tokens:  26738688000 | elapsed time per iteration (s): 0.37 | learning rate: 1.659E-04 | global batch size:   256 | lm loss: 3.337936E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.798 | TFLOPs: 32.43 |
7:  iteration    51100/  173500 | consumed samples:     13081600 | consumed tokens:  26791116800 | elapsed time per iteration (s): 0.37 | learning rate: 1.657E-04 | global batch size:   256 | lm loss: 3.337552E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.907 | TFLOPs: 32.39 |
7:  iteration    51200/  173500 | consumed samples:     13107200 | consumed tokens:  26843545600 | elapsed time per iteration (s): 0.37 | learning rate: 1.656E-04 | global batch size:   256 | lm loss: 3.334708E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.570 | TFLOPs: 32.33 |
7:  iteration    51300/  173500 | consumed samples:     13132800 | consumed tokens:  26895974400 | elapsed time per iteration (s): 0.37 | learning rate: 1.655E-04 | global batch size:   256 | lm loss: 3.332658E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.823 | TFLOPs: 32.39 |
7:  iteration    51400/  173500 | consumed samples:     13158400 | consumed tokens:  26948403200 | elapsed time per iteration (s): 0.37 | learning rate: 1.654E-04 | global batch size:   256 | lm loss: 3.334875E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.688 | TFLOPs: 32.38 |
7:  iteration    51500/  173500 | consumed samples:     13184000 | consumed tokens:  27000832000 | elapsed time per iteration (s): 0.37 | learning rate: 1.652E-04 | global batch size:   256 | lm loss: 3.336102E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.798 | TFLOPs: 32.38 |
7:  iteration    51600/  173500 | consumed samples:     13209600 | consumed tokens:  27053260800 | elapsed time per iteration (s): 0.37 | learning rate: 1.651E-04 | global batch size:   256 | lm loss: 3.330878E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.106 | TFLOPs: 32.26 |
7:  iteration    51700/  173500 | consumed samples:     13235200 | consumed tokens:  27105689600 | elapsed time per iteration (s): 0.37 | learning rate: 1.650E-04 | global batch size:   256 | lm loss: 3.331620E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.145 | TFLOPs: 32.31 |
7:  iteration    51800/  173500 | consumed samples:     13260800 | consumed tokens:  27158118400 | elapsed time per iteration (s): 0.37 | learning rate: 1.648E-04 | global batch size:   256 | lm loss: 3.331941E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.717 | TFLOPs: 32.33 |
7:  iteration    51900/  173500 | consumed samples:     13286400 | consumed tokens:  27210547200 | elapsed time per iteration (s): 0.37 | learning rate: 1.647E-04 | global batch size:   256 | lm loss: 3.331845E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.098 | TFLOPs: 32.30 |
0: [2023-03-17 04:45:59,944] [INFO] [logging.py:68:log_dist] [Rank 0] step=52000, skipped=0, lr=[0.00016457056203724818, 0.00016457056203724818, 0.00016457056203724818], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    52000/  173500 | consumed samples:     13312000 | consumed tokens:  27262976000 | elapsed time per iteration (s): 0.37 | learning rate: 1.646E-04 | global batch size:   256 | lm loss: 3.338719E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.318 | TFLOPs: 32.31 |
0: steps: 52000 loss: 3.3555 iter time (s): 0.368 samples/sec: 695.537
7:  iteration    52100/  173500 | consumed samples:     13337600 | consumed tokens:  27315404800 | elapsed time per iteration (s): 0.37 | learning rate: 1.644E-04 | global batch size:   256 | lm loss: 3.334371E+00 | grad norm: 0.199 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.878 | TFLOPs: 32.29 |
7:  iteration    52200/  173500 | consumed samples:     13363200 | consumed tokens:  27367833600 | elapsed time per iteration (s): 0.37 | learning rate: 1.643E-04 | global batch size:   256 | lm loss: 3.334156E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.109 | TFLOPs: 32.35 |
7:  iteration    52300/  173500 | consumed samples:     13388800 | consumed tokens:  27420262400 | elapsed time per iteration (s): 0.37 | learning rate: 1.642E-04 | global batch size:   256 | lm loss: 3.334227E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.726 | TFLOPs: 32.38 |
7:  iteration    52400/  173500 | consumed samples:     13414400 | consumed tokens:  27472691200 | elapsed time per iteration (s): 0.37 | learning rate: 1.640E-04 | global batch size:   256 | lm loss: 3.330439E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.777 | TFLOPs: 32.15 |
7:  iteration    52500/  173500 | consumed samples:     13440000 | consumed tokens:  27525120000 | elapsed time per iteration (s): 0.37 | learning rate: 1.639E-04 | global batch size:   256 | lm loss: 3.330178E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.179 | TFLOPs: 32.17 |
7:  iteration    52600/  173500 | consumed samples:     13465600 | consumed tokens:  27577548800 | elapsed time per iteration (s): 0.37 | learning rate: 1.638E-04 | global batch size:   256 | lm loss: 3.335356E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.258 | TFLOPs: 32.13 |
7:  iteration    52700/  173500 | consumed samples:     13491200 | consumed tokens:  27629977600 | elapsed time per iteration (s): 0.37 | learning rate: 1.636E-04 | global batch size:   256 | lm loss: 3.337205E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.658 | TFLOPs: 32.28 |
7:  iteration    52800/  173500 | consumed samples:     13516800 | consumed tokens:  27682406400 | elapsed time per iteration (s): 0.37 | learning rate: 1.635E-04 | global batch size:   256 | lm loss: 3.336209E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.970 | TFLOPs: 32.25 |
7:  iteration    52900/  173500 | consumed samples:     13542400 | consumed tokens:  27734835200 | elapsed time per iteration (s): 0.37 | learning rate: 1.634E-04 | global batch size:   256 | lm loss: 3.335555E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.239 | TFLOPs: 32.36 |
7:  iteration    53000/  173500 | consumed samples:     13568000 | consumed tokens:  27787264000 | elapsed time per iteration (s): 0.37 | learning rate: 1.633E-04 | global batch size:   256 | lm loss: 3.331373E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.247 | TFLOPs: 32.26 |
7:  iteration    53100/  173500 | consumed samples:     13593600 | consumed tokens:  27839692800 | elapsed time per iteration (s): 0.37 | learning rate: 1.631E-04 | global batch size:   256 | lm loss: 3.330514E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.807 | TFLOPs: 32.34 |
7:  iteration    53200/  173500 | consumed samples:     13619200 | consumed tokens:  27892121600 | elapsed time per iteration (s): 0.37 | learning rate: 1.630E-04 | global batch size:   256 | lm loss: 3.331394E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.697 | TFLOPs: 32.38 |
7:  iteration    53300/  173500 | consumed samples:     13644800 | consumed tokens:  27944550400 | elapsed time per iteration (s): 0.37 | learning rate: 1.629E-04 | global batch size:   256 | lm loss: 3.331273E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.367 | TFLOPs: 32.36 |
7:  iteration    53400/  173500 | consumed samples:     13670400 | consumed tokens:  27996979200 | elapsed time per iteration (s): 0.37 | learning rate: 1.627E-04 | global batch size:   256 | lm loss: 3.330431E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.875 | TFLOPs: 32.34 |
7:  iteration    53500/  173500 | consumed samples:     13696000 | consumed tokens:  28049408000 | elapsed time per iteration (s): 0.37 | learning rate: 1.626E-04 | global batch size:   256 | lm loss: 3.332549E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.052 | TFLOPs: 32.30 |
7:  iteration    53600/  173500 | consumed samples:     13721600 | consumed tokens:  28101836800 | elapsed time per iteration (s): 0.37 | learning rate: 1.625E-04 | global batch size:   256 | lm loss: 3.329965E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.477 | TFLOPs: 32.18 |
7:  iteration    53700/  173500 | consumed samples:     13747200 | consumed tokens:  28154265600 | elapsed time per iteration (s): 0.37 | learning rate: 1.623E-04 | global batch size:   256 | lm loss: 3.333279E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.781 | TFLOPs: 32.34 |
7:  iteration    53800/  173500 | consumed samples:     13772800 | consumed tokens:  28206694400 | elapsed time per iteration (s): 0.37 | learning rate: 1.622E-04 | global batch size:   256 | lm loss: 3.325459E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.831 | TFLOPs: 32.34 |
7:  iteration    53900/  173500 | consumed samples:     13798400 | consumed tokens:  28259123200 | elapsed time per iteration (s): 0.37 | learning rate: 1.621E-04 | global batch size:   256 | lm loss: 3.328750E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.005 | TFLOPs: 32.35 |
0: [2023-03-17 04:58:19,910] [INFO] [logging.py:68:log_dist] [Rank 0] step=54000, skipped=0, lr=[0.00016191666237869197, 0.00016191666237869197, 0.00016191666237869197], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    54000/  173500 | consumed samples:     13824000 | consumed tokens:  28311552000 | elapsed time per iteration (s): 0.37 | learning rate: 1.619E-04 | global batch size:   256 | lm loss: 3.329247E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.647 | TFLOPs: 32.38 |
0: steps: 54000 loss: 3.3423 iter time (s): 0.368 samples/sec: 695.293
7:  iteration    54100/  173500 | consumed samples:     13849600 | consumed tokens:  28363980800 | elapsed time per iteration (s): 0.37 | learning rate: 1.618E-04 | global batch size:   256 | lm loss: 3.330391E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.604 | TFLOPs: 32.37 |
7:  iteration    54200/  173500 | consumed samples:     13875200 | consumed tokens:  28416409600 | elapsed time per iteration (s): 0.37 | learning rate: 1.616E-04 | global batch size:   256 | lm loss: 3.326093E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.693 | TFLOPs: 32.38 |
7:  iteration    54300/  173500 | consumed samples:     13900800 | consumed tokens:  28468838400 | elapsed time per iteration (s): 0.37 | learning rate: 1.615E-04 | global batch size:   256 | lm loss: 3.328355E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.559 | TFLOPs: 32.33 |
7:  iteration    54400/  173500 | consumed samples:     13926400 | consumed tokens:  28521267200 | elapsed time per iteration (s): 0.37 | learning rate: 1.614E-04 | global batch size:   256 | lm loss: 3.330898E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.900 | TFLOPs: 32.34 |
7:  iteration    54500/  173500 | consumed samples:     13952000 | consumed tokens:  28573696000 | elapsed time per iteration (s): 0.37 | learning rate: 1.612E-04 | global batch size:   256 | lm loss: 3.329456E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.022 | TFLOPs: 32.35 |
7:  iteration    54600/  173500 | consumed samples:     13977600 | consumed tokens:  28626124800 | elapsed time per iteration (s): 0.37 | learning rate: 1.611E-04 | global batch size:   256 | lm loss: 3.328009E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.872 | TFLOPs: 32.34 |
7:  iteration    54700/  173500 | consumed samples:     14003200 | consumed tokens:  28678553600 | elapsed time per iteration (s): 0.37 | learning rate: 1.610E-04 | global batch size:   256 | lm loss: 3.326741E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.777 | TFLOPs: 32.38 |
7:  iteration    54800/  173500 | consumed samples:     14028800 | consumed tokens:  28730982400 | elapsed time per iteration (s): 0.37 | learning rate: 1.608E-04 | global batch size:   256 | lm loss: 3.332727E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.111 | TFLOPs: 32.35 |
7:  iteration    54900/  173500 | consumed samples:     14054400 | consumed tokens:  28783411200 | elapsed time per iteration (s): 0.37 | learning rate: 1.607E-04 | global batch size:   256 | lm loss: 3.330431E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.438 | TFLOPs: 32.37 |
7:  iteration    55000/  173500 | consumed samples:     14080000 | consumed tokens:  28835840000 | elapsed time per iteration (s): 0.37 | learning rate: 1.606E-04 | global batch size:   256 | lm loss: 3.327103E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.551 | TFLOPs: 32.37 |
7:  iteration    55100/  173500 | consumed samples:     14105600 | consumed tokens:  28888268800 | elapsed time per iteration (s): 0.37 | learning rate: 1.604E-04 | global batch size:   256 | lm loss: 3.329641E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.339 | TFLOPs: 32.36 |
7:  iteration    55200/  173500 | consumed samples:     14131200 | consumed tokens:  28940697600 | elapsed time per iteration (s): 0.37 | learning rate: 1.603E-04 | global batch size:   256 | lm loss: 3.329044E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.278 | TFLOPs: 32.36 |
7:  iteration    55300/  173500 | consumed samples:     14156800 | consumed tokens:  28993126400 | elapsed time per iteration (s): 0.37 | learning rate: 1.602E-04 | global batch size:   256 | lm loss: 3.327745E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.606 | TFLOPs: 32.37 |
7:  iteration    55400/  173500 | consumed samples:     14182400 | consumed tokens:  29045555200 | elapsed time per iteration (s): 0.37 | learning rate: 1.600E-04 | global batch size:   256 | lm loss: 3.331827E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.789 | TFLOPs: 32.38 |
7:  iteration    55500/  173500 | consumed samples:     14208000 | consumed tokens:  29097984000 | elapsed time per iteration (s): 0.53 | learning rate: 1.599E-04 | global batch size:   256 | lm loss: 3.324987E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 478.831 | TFLOPs: 22.35 |
7:  iteration    55600/  173500 | consumed samples:     14233600 | consumed tokens:  29150412800 | elapsed time per iteration (s): 0.37 | learning rate: 1.597E-04 | global batch size:   256 | lm loss: 3.325855E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.578 | TFLOPs: 32.42 |
7:  iteration    55700/  173500 | consumed samples:     14259200 | consumed tokens:  29202841600 | elapsed time per iteration (s): 0.37 | learning rate: 1.596E-04 | global batch size:   256 | lm loss: 3.330629E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.477 | TFLOPs: 32.32 |
7:  iteration    55800/  173500 | consumed samples:     14284800 | consumed tokens:  29255270400 | elapsed time per iteration (s): 0.37 | learning rate: 1.595E-04 | global batch size:   256 | lm loss: 3.328445E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.011 | TFLOPs: 32.39 |
7:  iteration    55900/  173500 | consumed samples:     14310400 | consumed tokens:  29307699200 | elapsed time per iteration (s): 0.37 | learning rate: 1.593E-04 | global batch size:   256 | lm loss: 3.330801E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.777 | TFLOPs: 32.38 |
0: [2023-03-17 05:10:54,811] [INFO] [logging.py:68:log_dist] [Rank 0] step=56000, skipped=0, lr=[0.0001591933009380588, 0.0001591933009380588, 0.0001591933009380588], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    56000/  173500 | consumed samples:     14336000 | consumed tokens:  29360128000 | elapsed time per iteration (s): 0.37 | learning rate: 1.592E-04 | global batch size:   256 | lm loss: 3.326139E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.864 | TFLOPs: 32.39 |
0: steps: 56000 loss: 3.2971 iter time (s): 0.375 samples/sec: 681.997
7:  iteration    56100/  173500 | consumed samples:     14361600 | consumed tokens:  29412556800 | elapsed time per iteration (s): 0.37 | learning rate: 1.591E-04 | global batch size:   256 | lm loss: 3.324270E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.362 | TFLOPs: 32.36 |
7:  iteration    56200/  173500 | consumed samples:     14387200 | consumed tokens:  29464985600 | elapsed time per iteration (s): 0.37 | learning rate: 1.589E-04 | global batch size:   256 | lm loss: 3.329599E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.762 | TFLOPs: 32.38 |
7:  iteration    56300/  173500 | consumed samples:     14412800 | consumed tokens:  29517414400 | elapsed time per iteration (s): 0.37 | learning rate: 1.588E-04 | global batch size:   256 | lm loss: 3.323502E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.984 | TFLOPs: 32.39 |
7:  iteration    56400/  173500 | consumed samples:     14438400 | consumed tokens:  29569843200 | elapsed time per iteration (s): 0.37 | learning rate: 1.586E-04 | global batch size:   256 | lm loss: 3.323917E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.595 | TFLOPs: 32.33 |
7:  iteration    56500/  173500 | consumed samples:     14464000 | consumed tokens:  29622272000 | elapsed time per iteration (s): 0.37 | learning rate: 1.585E-04 | global batch size:   256 | lm loss: 3.324552E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.912 | TFLOPs: 32.39 |
7:  iteration    56600/  173500 | consumed samples:     14489600 | consumed tokens:  29674700800 | elapsed time per iteration (s): 0.37 | learning rate: 1.584E-04 | global batch size:   256 | lm loss: 3.325466E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.891 | TFLOPs: 32.39 |
7:  iteration    56700/  173500 | consumed samples:     14515200 | consumed tokens:  29727129600 | elapsed time per iteration (s): 0.37 | learning rate: 1.582E-04 | global batch size:   256 | lm loss: 3.327299E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.858 | TFLOPs: 32.39 |
7:  iteration    56800/  173500 | consumed samples:     14540800 | consumed tokens:  29779558400 | elapsed time per iteration (s): 0.37 | learning rate: 1.581E-04 | global batch size:   256 | lm loss: 3.329081E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.685 | TFLOPs: 32.38 |
7:  iteration    56900/  173500 | consumed samples:     14566400 | consumed tokens:  29831987200 | elapsed time per iteration (s): 0.37 | learning rate: 1.579E-04 | global batch size:   256 | lm loss: 3.322562E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.317 | TFLOPs: 32.36 |
7:  iteration    57000/  173500 | consumed samples:     14592000 | consumed tokens:  29884416000 | elapsed time per iteration (s): 0.37 | learning rate: 1.578E-04 | global batch size:   256 | lm loss: 3.328902E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.838 | TFLOPs: 32.39 |
7:  iteration    57100/  173500 | consumed samples:     14617600 | consumed tokens:  29936844800 | elapsed time per iteration (s): 0.37 | learning rate: 1.577E-04 | global batch size:   256 | lm loss: 3.322817E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.736 | TFLOPs: 32.38 |
7:  iteration    57200/  173500 | consumed samples:     14643200 | consumed tokens:  29989273600 | elapsed time per iteration (s): 0.37 | learning rate: 1.575E-04 | global batch size:   256 | lm loss: 3.322450E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.085 | TFLOPs: 32.40 |
7:  iteration    57300/  173500 | consumed samples:     14668800 | consumed tokens:  30041702400 | elapsed time per iteration (s): 0.37 | learning rate: 1.574E-04 | global batch size:   256 | lm loss: 3.321382E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.830 | TFLOPs: 32.25 |
7:  iteration    57400/  173500 | consumed samples:     14694400 | consumed tokens:  30094131200 | elapsed time per iteration (s): 0.37 | learning rate: 1.572E-04 | global batch size:   256 | lm loss: 3.321409E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.612 | TFLOPs: 32.38 |
7:  iteration    57500/  173500 | consumed samples:     14720000 | consumed tokens:  30146560000 | elapsed time per iteration (s): 0.37 | learning rate: 1.571E-04 | global batch size:   256 | lm loss: 3.325367E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.823 | TFLOPs: 32.29 |
7:  iteration    57600/  173500 | consumed samples:     14745600 | consumed tokens:  30198988800 | elapsed time per iteration (s): 0.37 | learning rate: 1.570E-04 | global batch size:   256 | lm loss: 3.326556E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.061 | TFLOPs: 32.30 |
7:  iteration    57700/  173500 | consumed samples:     14771200 | consumed tokens:  30251417600 | elapsed time per iteration (s): 0.37 | learning rate: 1.568E-04 | global batch size:   256 | lm loss: 3.320583E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.190 | TFLOPs: 32.40 |
7:  iteration    57800/  173500 | consumed samples:     14796800 | consumed tokens:  30303846400 | elapsed time per iteration (s): 0.37 | learning rate: 1.567E-04 | global batch size:   256 | lm loss: 3.324065E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.648 | TFLOPs: 32.38 |
7:  iteration    57900/  173500 | consumed samples:     14822400 | consumed tokens:  30356275200 | elapsed time per iteration (s): 0.37 | learning rate: 1.565E-04 | global batch size:   256 | lm loss: 3.322610E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.332 | TFLOPs: 32.36 |
0: [2023-03-17 05:23:13,277] [INFO] [logging.py:68:log_dist] [Rank 0] step=58000, skipped=0, lr=[0.00015640412143068475, 0.00015640412143068475, 0.00015640412143068475], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    58000/  173500 | consumed samples:     14848000 | consumed tokens:  30408704000 | elapsed time per iteration (s): 0.37 | learning rate: 1.564E-04 | global batch size:   256 | lm loss: 3.322589E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.080 | TFLOPs: 32.35 |
0: steps: 58000 loss: 3.3004 iter time (s): 0.367 samples/sec: 697.166
7:  iteration    58100/  173500 | consumed samples:     14873600 | consumed tokens:  30461132800 | elapsed time per iteration (s): 0.37 | learning rate: 1.563E-04 | global batch size:   256 | lm loss: 3.319429E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.721 | TFLOPs: 32.38 |
7:  iteration    58200/  173500 | consumed samples:     14899200 | consumed tokens:  30513561600 | elapsed time per iteration (s): 0.37 | learning rate: 1.561E-04 | global batch size:   256 | lm loss: 3.322419E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.088 | TFLOPs: 32.40 |
7:  iteration    58300/  173500 | consumed samples:     14924800 | consumed tokens:  30565990400 | elapsed time per iteration (s): 0.37 | learning rate: 1.560E-04 | global batch size:   256 | lm loss: 3.324950E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.205 | TFLOPs: 32.40 |
7:  iteration    58400/  173500 | consumed samples:     14950400 | consumed tokens:  30618419200 | elapsed time per iteration (s): 0.37 | learning rate: 1.558E-04 | global batch size:   256 | lm loss: 3.320234E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.406 | TFLOPs: 32.41 |
7:  iteration    58500/  173500 | consumed samples:     14976000 | consumed tokens:  30670848000 | elapsed time per iteration (s): 0.37 | learning rate: 1.557E-04 | global batch size:   256 | lm loss: 3.318896E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.503 | TFLOPs: 32.42 |
7:  iteration    58600/  173500 | consumed samples:     15001600 | consumed tokens:  30723276800 | elapsed time per iteration (s): 0.37 | learning rate: 1.556E-04 | global batch size:   256 | lm loss: 3.319357E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.558 | TFLOPs: 32.42 |
7:  iteration    58700/  173500 | consumed samples:     15027200 | consumed tokens:  30775705600 | elapsed time per iteration (s): 0.37 | learning rate: 1.554E-04 | global batch size:   256 | lm loss: 3.319598E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.151 | TFLOPs: 32.40 |
7:  iteration    58800/  173500 | consumed samples:     15052800 | consumed tokens:  30828134400 | elapsed time per iteration (s): 0.37 | learning rate: 1.553E-04 | global batch size:   256 | lm loss: 3.321166E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.808 | TFLOPs: 32.38 |
7:  iteration    58900/  173500 | consumed samples:     15078400 | consumed tokens:  30880563200 | elapsed time per iteration (s): 0.37 | learning rate: 1.551E-04 | global batch size:   256 | lm loss: 3.320921E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.544 | TFLOPs: 32.37 |
7:  iteration    59000/  173500 | consumed samples:     15104000 | consumed tokens:  30932992000 | elapsed time per iteration (s): 0.37 | learning rate: 1.550E-04 | global batch size:   256 | lm loss: 3.322201E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.135 | TFLOPs: 32.40 |
7:  iteration    59100/  173500 | consumed samples:     15129600 | consumed tokens:  30985420800 | elapsed time per iteration (s): 0.37 | learning rate: 1.548E-04 | global batch size:   256 | lm loss: 3.319790E+00 | grad norm: 0.247 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.193 | TFLOPs: 32.40 |
7:  iteration    59200/  173500 | consumed samples:     15155200 | consumed tokens:  31037849600 | elapsed time per iteration (s): 0.37 | learning rate: 1.547E-04 | global batch size:   256 | lm loss: 3.322111E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.668 | TFLOPs: 31.96 |
7:  iteration    59300/  173500 | consumed samples:     15180800 | consumed tokens:  31090278400 | elapsed time per iteration (s): 0.37 | learning rate: 1.546E-04 | global batch size:   256 | lm loss: 3.322382E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.406 | TFLOPs: 31.90 |
7:  iteration    59400/  173500 | consumed samples:     15206400 | consumed tokens:  31142707200 | elapsed time per iteration (s): 0.38 | learning rate: 1.544E-04 | global batch size:   256 | lm loss: 3.318034E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.105 | TFLOPs: 31.70 |
7:  iteration    59500/  173500 | consumed samples:     15232000 | consumed tokens:  31195136000 | elapsed time per iteration (s): 0.38 | learning rate: 1.543E-04 | global batch size:   256 | lm loss: 3.322572E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 676.098 | TFLOPs: 31.56 |
7:  iteration    59600/  173500 | consumed samples:     15257600 | consumed tokens:  31247564800 | elapsed time per iteration (s): 0.38 | learning rate: 1.541E-04 | global batch size:   256 | lm loss: 3.320048E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 671.551 | TFLOPs: 31.35 |
7:  iteration    59700/  173500 | consumed samples:     15283200 | consumed tokens:  31299993600 | elapsed time per iteration (s): 0.38 | learning rate: 1.540E-04 | global batch size:   256 | lm loss: 3.318021E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.629 | TFLOPs: 31.40 |
7:  iteration    59800/  173500 | consumed samples:     15308800 | consumed tokens:  31352422400 | elapsed time per iteration (s): 0.38 | learning rate: 1.538E-04 | global batch size:   256 | lm loss: 3.321158E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 676.589 | TFLOPs: 31.58 |
7:  iteration    59900/  173500 | consumed samples:     15334400 | consumed tokens:  31404851200 | elapsed time per iteration (s): 0.38 | learning rate: 1.537E-04 | global batch size:   256 | lm loss: 3.315964E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 677.058 | TFLOPs: 31.60 |
0: [2023-03-17 05:35:38,992] [INFO] [logging.py:68:log_dist] [Rank 0] step=60000, skipped=0, lr=[0.00015355285563304073, 0.00015355285563304073, 0.00015355285563304073], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    60000/  173500 | consumed samples:     15360000 | consumed tokens:  31457280000 | elapsed time per iteration (s): 0.38 | learning rate: 1.536E-04 | global batch size:   256 | lm loss: 3.319155E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 677.567 | TFLOPs: 31.63 |
0: steps: 60000 loss: 3.3682 iter time (s): 0.371 samples/sec: 690.349
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 60000 | lm loss value: 3.396574E+00 | lm loss PPL: 2.986162E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   60000 to checkpoints_146m91b400m
0: [2023-03-17 05:35:39,119] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step60000 is begin to save!
0: [2023-03-17 05:35:39,124] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_01-model_00-model_states.pt...
0: [2023-03-17 05:35:39,225] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_01-model_00-model_states.pt.
0: [2023-03-17 05:35:39,225] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_03-model_00-model_states.pt...
0: [2023-03-17 05:35:39,240] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_03-model_00-model_states.pt.
0: [2023-03-17 05:35:39,241] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_04-model_00-model_states.pt...
0: [2023-03-17 05:35:39,255] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_04-model_00-model_states.pt.
0: [2023-03-17 05:35:39,256] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_05-model_00-model_states.pt...
0: [2023-03-17 05:35:39,270] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_05-model_00-model_states.pt.
0: [2023-03-17 05:35:39,271] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_06-model_00-model_states.pt...
0: [2023-03-17 05:35:39,285] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_06-model_00-model_states.pt.
0: [2023-03-17 05:35:39,286] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_07-model_00-model_states.pt...
0: [2023-03-17 05:35:39,300] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_07-model_00-model_states.pt.
0: [2023-03-17 05:35:39,301] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_08-model_00-model_states.pt...
0: [2023-03-17 05:35:39,315] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_08-model_00-model_states.pt.
0: [2023-03-17 05:35:39,316] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_09-model_00-model_states.pt...
0: [2023-03-17 05:35:39,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_09-model_00-model_states.pt.
0: [2023-03-17 05:35:39,330] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_10-model_00-model_states.pt...
0: [2023-03-17 05:35:39,345] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_10-model_00-model_states.pt.
0: [2023-03-17 05:35:39,345] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_11-model_00-model_states.pt...
0: [2023-03-17 05:35:39,360] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_11-model_00-model_states.pt.
0: [2023-03-17 05:35:39,360] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_12-model_00-model_states.pt...
0: [2023-03-17 05:35:39,375] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_12-model_00-model_states.pt.
0: [2023-03-17 05:35:39,375] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_13-model_00-model_states.pt...
0: [2023-03-17 05:35:39,390] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_13-model_00-model_states.pt.
0: [2023-03-17 05:35:39,390] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_14-model_00-model_states.pt...
0: [2023-03-17 05:35:39,405] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_14-model_00-model_states.pt.
0: [2023-03-17 05:35:39,405] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_15-model_00-model_states.pt...
0: [2023-03-17 05:35:39,420] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_15-model_00-model_states.pt.
0: [2023-03-17 05:35:39,420] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_16-model_00-model_states.pt...
0: [2023-03-17 05:35:39,435] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_16-model_00-model_states.pt.
0: [2023-03-17 05:35:39,435] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_17-model_00-model_states.pt...
0: [2023-03-17 05:35:39,450] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_17-model_00-model_states.pt.
0: [2023-03-17 05:35:39,450] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/layer_19-model_00-model_states.pt...
0: [2023-03-17 05:35:39,451] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/layer_19-model_00-model_states.pt.
0: [2023-03-17 05:35:39,452] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step60000/mp_rank_00_model_states.pt
0: [2023-03-17 05:35:39,452] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/mp_rank_00_model_states.pt...
0: [2023-03-17 05:35:39,456] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/mp_rank_00_model_states.pt.
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
3: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 05:35:39,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
0: [2023-03-17 05:35:39,503] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,510] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,510] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,510] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,510] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,510] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,513] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
3: [2023-03-17 05:35:39,516] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 05:35:39,517] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 05:35:39,517] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 05:35:39,517] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,517] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,517] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,517] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,530] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,531] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
6: [2023-03-17 05:35:39,532] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 05:35:39,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,533] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,533] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 05:35:39,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,534] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
7: [2023-03-17 05:35:39,534] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
1: [2023-03-17 05:35:39,536] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 05:35:39,537] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 05:35:39,537] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0: [2023-03-17 05:35:39,537] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 05:35:39,537] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
4: [2023-03-17 05:35:39,544] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
5: [2023-03-17 05:35:39,550] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
2: [2023-03-17 05:35:39,557] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now!
0:   successfully saved checkpoint at iteration   60000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 447.96
7:  iteration    60100/  173500 | consumed samples:     15385600 | consumed tokens:  31509708800 | elapsed time per iteration (s): 0.38 | learning rate: 1.534E-04 | global batch size:   256 | lm loss: 3.315382E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 665.028 | TFLOPs: 31.04 |
7:  iteration    60200/  173500 | consumed samples:     15411200 | consumed tokens:  31562137600 | elapsed time per iteration (s): 0.37 | learning rate: 1.533E-04 | global batch size:   256 | lm loss: 3.317213E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.023 | TFLOPs: 32.07 |
7:  iteration    60300/  173500 | consumed samples:     15436800 | consumed tokens:  31614566400 | elapsed time per iteration (s): 0.37 | learning rate: 1.531E-04 | global batch size:   256 | lm loss: 3.315157E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.123 | TFLOPs: 32.26 |
7:  iteration    60400/  173500 | consumed samples:     15462400 | consumed tokens:  31666995200 | elapsed time per iteration (s): 0.37 | learning rate: 1.530E-04 | global batch size:   256 | lm loss: 3.315771E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.238 | TFLOPs: 31.98 |
7:  iteration    60500/  173500 | consumed samples:     15488000 | consumed tokens:  31719424000 | elapsed time per iteration (s): 0.37 | learning rate: 1.528E-04 | global batch size:   256 | lm loss: 3.314309E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.828 | TFLOPs: 32.11 |
7:  iteration    60600/  173500 | consumed samples:     15513600 | consumed tokens:  31771852800 | elapsed time per iteration (s): 0.37 | learning rate: 1.527E-04 | global batch size:   256 | lm loss: 3.317643E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.642 | TFLOPs: 32.05 |
7:  iteration    60700/  173500 | consumed samples:     15539200 | consumed tokens:  31824281600 | elapsed time per iteration (s): 0.37 | learning rate: 1.525E-04 | global batch size:   256 | lm loss: 3.318480E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.154 | TFLOPs: 32.26 |
7:  iteration    60800/  173500 | consumed samples:     15564800 | consumed tokens:  31876710400 | elapsed time per iteration (s): 0.37 | learning rate: 1.524E-04 | global batch size:   256 | lm loss: 3.318091E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.826 | TFLOPs: 32.01 |
7:  iteration    60900/  173500 | consumed samples:     15590400 | consumed tokens:  31929139200 | elapsed time per iteration (s): 0.37 | learning rate: 1.523E-04 | global batch size:   256 | lm loss: 3.314496E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.958 | TFLOPs: 31.97 |
7:  iteration    61000/  173500 | consumed samples:     15616000 | consumed tokens:  31981568000 | elapsed time per iteration (s): 0.37 | learning rate: 1.521E-04 | global batch size:   256 | lm loss: 3.313748E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.235 | TFLOPs: 32.17 |
7:  iteration    61100/  173500 | consumed samples:     15641600 | consumed tokens:  32033996800 | elapsed time per iteration (s): 0.37 | learning rate: 1.520E-04 | global batch size:   256 | lm loss: 3.315529E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.792 | TFLOPs: 32.01 |
7:  iteration    61200/  173500 | consumed samples:     15667200 | consumed tokens:  32086425600 | elapsed time per iteration (s): 0.37 | learning rate: 1.518E-04 | global batch size:   256 | lm loss: 3.316752E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.804 | TFLOPs: 32.15 |
7:  iteration    61300/  173500 | consumed samples:     15692800 | consumed tokens:  32138854400 | elapsed time per iteration (s): 0.37 | learning rate: 1.517E-04 | global batch size:   256 | lm loss: 3.320029E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.618 | TFLOPs: 32.14 |
7:  iteration    61400/  173500 | consumed samples:     15718400 | consumed tokens:  32191283200 | elapsed time per iteration (s): 0.37 | learning rate: 1.515E-04 | global batch size:   256 | lm loss: 3.312858E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.306 | TFLOPs: 32.03 |
7:  iteration    61500/  173500 | consumed samples:     15744000 | consumed tokens:  32243712000 | elapsed time per iteration (s): 0.37 | learning rate: 1.514E-04 | global batch size:   256 | lm loss: 3.314777E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.037 | TFLOPs: 32.30 |
7:  iteration    61600/  173500 | consumed samples:     15769600 | consumed tokens:  32296140800 | elapsed time per iteration (s): 0.37 | learning rate: 1.512E-04 | global batch size:   256 | lm loss: 3.315212E+00 | grad norm: 0.189 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.199 | TFLOPs: 32.31 |
7:  iteration    61700/  173500 | consumed samples:     15795200 | consumed tokens:  32348569600 | elapsed time per iteration (s): 0.37 | learning rate: 1.511E-04 | global batch size:   256 | lm loss: 3.316159E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.161 | TFLOPs: 32.21 |
7:  iteration    61800/  173500 | consumed samples:     15820800 | consumed tokens:  32400998400 | elapsed time per iteration (s): 0.37 | learning rate: 1.509E-04 | global batch size:   256 | lm loss: 3.314585E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.069 | TFLOPs: 32.12 |
7:  iteration    61900/  173500 | consumed samples:     15846400 | consumed tokens:  32453427200 | elapsed time per iteration (s): 0.37 | learning rate: 1.508E-04 | global batch size:   256 | lm loss: 3.311006E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.047 | TFLOPs: 32.12 |
0: [2023-03-17 05:48:04,120] [INFO] [logging.py:68:log_dist] [Rank 0] step=62000, skipped=0, lr=[0.00015064331838981058, 0.00015064331838981058, 0.00015064331838981058], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    62000/  173500 | consumed samples:     15872000 | consumed tokens:  32505856000 | elapsed time per iteration (s): 0.37 | learning rate: 1.506E-04 | global batch size:   256 | lm loss: 3.316156E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.431 | TFLOPs: 32.18 |
0: steps: 62000 loss: 3.3377 iter time (s): 0.371 samples/sec: 690.256
7:  iteration    62100/  173500 | consumed samples:     15897600 | consumed tokens:  32558284800 | elapsed time per iteration (s): 0.37 | learning rate: 1.505E-04 | global batch size:   256 | lm loss: 3.310527E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.098 | TFLOPs: 32.07 |
7:  iteration    62200/  173500 | consumed samples:     15923200 | consumed tokens:  32610713600 | elapsed time per iteration (s): 0.37 | learning rate: 1.503E-04 | global batch size:   256 | lm loss: 3.314499E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.102 | TFLOPs: 32.21 |
7:  iteration    62300/  173500 | consumed samples:     15948800 | consumed tokens:  32663142400 | elapsed time per iteration (s): 0.37 | learning rate: 1.502E-04 | global batch size:   256 | lm loss: 3.315598E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.296 | TFLOPs: 32.27 |
7:  iteration    62400/  173500 | consumed samples:     15974400 | consumed tokens:  32715571200 | elapsed time per iteration (s): 0.37 | learning rate: 1.501E-04 | global batch size:   256 | lm loss: 3.314664E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.536 | TFLOPs: 32.19 |
7:  iteration    62500/  173500 | consumed samples:     16000000 | consumed tokens:  32768000000 | elapsed time per iteration (s): 0.37 | learning rate: 1.499E-04 | global batch size:   256 | lm loss: 3.311529E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.269 | TFLOPs: 31.89 |
7:  iteration    62600/  173500 | consumed samples:     16025600 | consumed tokens:  32820428800 | elapsed time per iteration (s): 0.37 | learning rate: 1.498E-04 | global batch size:   256 | lm loss: 3.314771E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.086 | TFLOPs: 32.21 |
7:  iteration    62700/  173500 | consumed samples:     16051200 | consumed tokens:  32872857600 | elapsed time per iteration (s): 0.37 | learning rate: 1.496E-04 | global batch size:   256 | lm loss: 3.311366E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.757 | TFLOPs: 32.34 |
7:  iteration    62800/  173500 | consumed samples:     16076800 | consumed tokens:  32925286400 | elapsed time per iteration (s): 0.37 | learning rate: 1.495E-04 | global batch size:   256 | lm loss: 3.309564E+00 | grad norm: 0.244 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.169 | TFLOPs: 32.35 |
7:  iteration    62900/  173500 | consumed samples:     16102400 | consumed tokens:  32977715200 | elapsed time per iteration (s): 0.37 | learning rate: 1.493E-04 | global batch size:   256 | lm loss: 3.314532E+00 | grad norm: 0.259 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.008 | TFLOPs: 32.07 |
7:  iteration    63000/  173500 | consumed samples:     16128000 | consumed tokens:  33030144000 | elapsed time per iteration (s): 0.37 | learning rate: 1.492E-04 | global batch size:   256 | lm loss: 3.312073E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.198 | TFLOPs: 32.22 |
7:  iteration    63100/  173500 | consumed samples:     16153600 | consumed tokens:  33082572800 | elapsed time per iteration (s): 0.37 | learning rate: 1.490E-04 | global batch size:   256 | lm loss: 3.310173E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.548 | TFLOPs: 32.09 |
7:  iteration    63200/  173500 | consumed samples:     16179200 | consumed tokens:  33135001600 | elapsed time per iteration (s): 0.37 | learning rate: 1.489E-04 | global batch size:   256 | lm loss: 3.313134E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.453 | TFLOPs: 32.32 |
7:  iteration    63300/  173500 | consumed samples:     16204800 | consumed tokens:  33187430400 | elapsed time per iteration (s): 0.37 | learning rate: 1.487E-04 | global batch size:   256 | lm loss: 3.313323E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.566 | TFLOPs: 32.37 |
7:  iteration    63400/  173500 | consumed samples:     16230400 | consumed tokens:  33239859200 | elapsed time per iteration (s): 0.37 | learning rate: 1.486E-04 | global batch size:   256 | lm loss: 3.309963E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.914 | TFLOPs: 32.30 |
7:  iteration    63500/  173500 | consumed samples:     16256000 | consumed tokens:  33292288000 | elapsed time per iteration (s): 0.37 | learning rate: 1.484E-04 | global batch size:   256 | lm loss: 3.313337E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.294 | TFLOPs: 32.41 |
7:  iteration    63600/  173500 | consumed samples:     16281600 | consumed tokens:  33344716800 | elapsed time per iteration (s): 0.37 | learning rate: 1.483E-04 | global batch size:   256 | lm loss: 3.310764E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.798 | TFLOPs: 32.43 |
7:  iteration    63700/  173500 | consumed samples:     16307200 | consumed tokens:  33397145600 | elapsed time per iteration (s): 0.37 | learning rate: 1.481E-04 | global batch size:   256 | lm loss: 3.313407E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.728 | TFLOPs: 32.43 |
7:  iteration    63800/  173500 | consumed samples:     16332800 | consumed tokens:  33449574400 | elapsed time per iteration (s): 0.37 | learning rate: 1.480E-04 | global batch size:   256 | lm loss: 3.315344E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.709 | TFLOPs: 32.43 |
7:  iteration    63900/  173500 | consumed samples:     16358400 | consumed tokens:  33502003200 | elapsed time per iteration (s): 0.37 | learning rate: 1.478E-04 | global batch size:   256 | lm loss: 3.308877E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.959 | TFLOPs: 32.30 |
0: [2023-03-17 06:00:24,857] [INFO] [logging.py:68:log_dist] [Rank 0] step=64000, skipped=0, lr=[0.0001476794025098283, 0.0001476794025098283, 0.0001476794025098283], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    64000/  173500 | consumed samples:     16384000 | consumed tokens:  33554432000 | elapsed time per iteration (s): 0.37 | learning rate: 1.477E-04 | global batch size:   256 | lm loss: 3.313734E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.879 | TFLOPs: 32.39 |
0: steps: 64000 loss: 3.3472 iter time (s): 0.369 samples/sec: 693.880
7:  iteration    64100/  173500 | consumed samples:     16409600 | consumed tokens:  33606860800 | elapsed time per iteration (s): 0.37 | learning rate: 1.475E-04 | global batch size:   256 | lm loss: 3.311430E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.387 | TFLOPs: 32.41 |
7:  iteration    64200/  173500 | consumed samples:     16435200 | consumed tokens:  33659289600 | elapsed time per iteration (s): 0.37 | learning rate: 1.474E-04 | global batch size:   256 | lm loss: 3.314964E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.705 | TFLOPs: 32.43 |
7:  iteration    64300/  173500 | consumed samples:     16460800 | consumed tokens:  33711718400 | elapsed time per iteration (s): 0.37 | learning rate: 1.472E-04 | global batch size:   256 | lm loss: 3.309182E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.723 | TFLOPs: 32.43 |
7:  iteration    64400/  173500 | consumed samples:     16486400 | consumed tokens:  33764147200 | elapsed time per iteration (s): 0.37 | learning rate: 1.471E-04 | global batch size:   256 | lm loss: 3.310246E+00 | grad norm: 0.200 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.833 | TFLOPs: 32.43 |
7:  iteration    64500/  173500 | consumed samples:     16512000 | consumed tokens:  33816576000 | elapsed time per iteration (s): 0.37 | learning rate: 1.469E-04 | global batch size:   256 | lm loss: 3.307413E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.993 | TFLOPs: 32.39 |
7:  iteration    64600/  173500 | consumed samples:     16537600 | consumed tokens:  33869004800 | elapsed time per iteration (s): 0.37 | learning rate: 1.468E-04 | global batch size:   256 | lm loss: 3.313589E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.727 | TFLOPs: 32.33 |
7:  iteration    64700/  173500 | consumed samples:     16563200 | consumed tokens:  33921433600 | elapsed time per iteration (s): 0.37 | learning rate: 1.466E-04 | global batch size:   256 | lm loss: 3.306772E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.817 | TFLOPs: 32.43 |
7:  iteration    64800/  173500 | consumed samples:     16588800 | consumed tokens:  33973862400 | elapsed time per iteration (s): 0.37 | learning rate: 1.465E-04 | global batch size:   256 | lm loss: 3.314012E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.660 | TFLOPs: 32.42 |
7:  iteration    64900/  173500 | consumed samples:     16614400 | consumed tokens:  34026291200 | elapsed time per iteration (s): 0.37 | learning rate: 1.463E-04 | global batch size:   256 | lm loss: 3.310314E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.375 | TFLOPs: 32.27 |
7:  iteration    65000/  173500 | consumed samples:     16640000 | consumed tokens:  34078720000 | elapsed time per iteration (s): 0.37 | learning rate: 1.462E-04 | global batch size:   256 | lm loss: 3.312580E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.587 | TFLOPs: 32.42 |
7:  iteration    65100/  173500 | consumed samples:     16665600 | consumed tokens:  34131148800 | elapsed time per iteration (s): 0.37 | learning rate: 1.460E-04 | global batch size:   256 | lm loss: 3.311990E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.825 | TFLOPs: 32.39 |
7:  iteration    65200/  173500 | consumed samples:     16691200 | consumed tokens:  34183577600 | elapsed time per iteration (s): 0.37 | learning rate: 1.459E-04 | global batch size:   256 | lm loss: 3.305869E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.705 | TFLOPs: 32.38 |
7:  iteration    65300/  173500 | consumed samples:     16716800 | consumed tokens:  34236006400 | elapsed time per iteration (s): 0.37 | learning rate: 1.457E-04 | global batch size:   256 | lm loss: 3.306133E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.657 | TFLOPs: 32.38 |
7:  iteration    65400/  173500 | consumed samples:     16742400 | consumed tokens:  34288435200 | elapsed time per iteration (s): 0.37 | learning rate: 1.456E-04 | global batch size:   256 | lm loss: 3.311733E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.200 | TFLOPs: 32.40 |
7:  iteration    65500/  173500 | consumed samples:     16768000 | consumed tokens:  34340864000 | elapsed time per iteration (s): 0.37 | learning rate: 1.454E-04 | global batch size:   256 | lm loss: 3.306813E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.422 | TFLOPs: 32.37 |
7:  iteration    65600/  173500 | consumed samples:     16793600 | consumed tokens:  34393292800 | elapsed time per iteration (s): 0.37 | learning rate: 1.453E-04 | global batch size:   256 | lm loss: 3.306375E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.665 | TFLOPs: 32.33 |
7:  iteration    65700/  173500 | consumed samples:     16819200 | consumed tokens:  34445721600 | elapsed time per iteration (s): 0.37 | learning rate: 1.451E-04 | global batch size:   256 | lm loss: 3.308405E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.264 | TFLOPs: 32.41 |
7:  iteration    65800/  173500 | consumed samples:     16844800 | consumed tokens:  34498150400 | elapsed time per iteration (s): 0.37 | learning rate: 1.450E-04 | global batch size:   256 | lm loss: 3.308539E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.928 | TFLOPs: 32.39 |
7:  iteration    65900/  173500 | consumed samples:     16870400 | consumed tokens:  34550579200 | elapsed time per iteration (s): 0.37 | learning rate: 1.448E-04 | global batch size:   256 | lm loss: 3.300935E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.065 | TFLOPs: 32.40 |
0: [2023-03-17 06:12:42,679] [INFO] [logging.py:68:log_dist] [Rank 0] step=66000, skipped=0, lr=[0.00014466507355770288, 0.00014466507355770288, 0.00014466507355770288], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    66000/  173500 | consumed samples:     16896000 | consumed tokens:  34603008000 | elapsed time per iteration (s): 0.37 | learning rate: 1.447E-04 | global batch size:   256 | lm loss: 3.307038E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.195 | TFLOPs: 32.40 |
0: steps: 66000 loss: 3.3181 iter time (s): 0.367 samples/sec: 696.655
7:  iteration    66100/  173500 | consumed samples:     16921600 | consumed tokens:  34655436800 | elapsed time per iteration (s): 0.37 | learning rate: 1.445E-04 | global batch size:   256 | lm loss: 3.305034E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.557 | TFLOPs: 32.37 |
7:  iteration    66200/  173500 | consumed samples:     16947200 | consumed tokens:  34707865600 | elapsed time per iteration (s): 0.37 | learning rate: 1.444E-04 | global batch size:   256 | lm loss: 3.304758E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.174 | TFLOPs: 32.40 |
7:  iteration    66300/  173500 | consumed samples:     16972800 | consumed tokens:  34760294400 | elapsed time per iteration (s): 0.37 | learning rate: 1.442E-04 | global batch size:   256 | lm loss: 3.306238E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.702 | TFLOPs: 32.38 |
7:  iteration    66400/  173500 | consumed samples:     16998400 | consumed tokens:  34812723200 | elapsed time per iteration (s): 0.37 | learning rate: 1.441E-04 | global batch size:   256 | lm loss: 3.310963E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.535 | TFLOPs: 32.42 |
7:  iteration    66500/  173500 | consumed samples:     17024000 | consumed tokens:  34865152000 | elapsed time per iteration (s): 0.37 | learning rate: 1.439E-04 | global batch size:   256 | lm loss: 3.303157E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.755 | TFLOPs: 32.43 |
7:  iteration    66600/  173500 | consumed samples:     17049600 | consumed tokens:  34917580800 | elapsed time per iteration (s): 0.37 | learning rate: 1.438E-04 | global batch size:   256 | lm loss: 3.302287E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.278 | TFLOPs: 32.41 |
7:  iteration    66700/  173500 | consumed samples:     17075200 | consumed tokens:  34970009600 | elapsed time per iteration (s): 0.37 | learning rate: 1.436E-04 | global batch size:   256 | lm loss: 3.299700E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.413 | TFLOPs: 32.41 |
7:  iteration    66800/  173500 | consumed samples:     17100800 | consumed tokens:  35022438400 | elapsed time per iteration (s): 0.37 | learning rate: 1.434E-04 | global batch size:   256 | lm loss: 3.305618E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.958 | TFLOPs: 32.39 |
7:  iteration    66900/  173500 | consumed samples:     17126400 | consumed tokens:  35074867200 | elapsed time per iteration (s): 0.37 | learning rate: 1.433E-04 | global batch size:   256 | lm loss: 3.301610E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.643 | TFLOPs: 32.42 |
7:  iteration    67000/  173500 | consumed samples:     17152000 | consumed tokens:  35127296000 | elapsed time per iteration (s): 0.37 | learning rate: 1.431E-04 | global batch size:   256 | lm loss: 3.303792E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.738 | TFLOPs: 32.43 |
7:  iteration    67100/  173500 | consumed samples:     17177600 | consumed tokens:  35179724800 | elapsed time per iteration (s): 0.37 | learning rate: 1.430E-04 | global batch size:   256 | lm loss: 3.305234E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.320 | TFLOPs: 32.41 |
7:  iteration    67200/  173500 | consumed samples:     17203200 | consumed tokens:  35232153600 | elapsed time per iteration (s): 0.37 | learning rate: 1.428E-04 | global batch size:   256 | lm loss: 3.305688E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.471 | TFLOPs: 32.42 |
7:  iteration    67300/  173500 | consumed samples:     17228800 | consumed tokens:  35284582400 | elapsed time per iteration (s): 0.37 | learning rate: 1.427E-04 | global batch size:   256 | lm loss: 3.304243E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.238 | TFLOPs: 32.40 |
7:  iteration    67400/  173500 | consumed samples:     17254400 | consumed tokens:  35337011200 | elapsed time per iteration (s): 0.37 | learning rate: 1.425E-04 | global batch size:   256 | lm loss: 3.298444E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.153 | TFLOPs: 32.40 |
7:  iteration    67500/  173500 | consumed samples:     17280000 | consumed tokens:  35389440000 | elapsed time per iteration (s): 0.37 | learning rate: 1.424E-04 | global batch size:   256 | lm loss: 3.306365E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.049 | TFLOPs: 32.40 |
7:  iteration    67600/  173500 | consumed samples:     17305600 | consumed tokens:  35441868800 | elapsed time per iteration (s): 0.37 | learning rate: 1.422E-04 | global batch size:   256 | lm loss: 3.299843E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.033 | TFLOPs: 32.39 |
7:  iteration    67700/  173500 | consumed samples:     17331200 | consumed tokens:  35494297600 | elapsed time per iteration (s): 0.37 | learning rate: 1.421E-04 | global batch size:   256 | lm loss: 3.302761E+00 | grad norm: 0.205 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.321 | TFLOPs: 32.41 |
7:  iteration    67800/  173500 | consumed samples:     17356800 | consumed tokens:  35546726400 | elapsed time per iteration (s): 0.37 | learning rate: 1.419E-04 | global batch size:   256 | lm loss: 3.298817E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.963 | TFLOPs: 32.34 |
7:  iteration    67900/  173500 | consumed samples:     17382400 | consumed tokens:  35599155200 | elapsed time per iteration (s): 0.37 | learning rate: 1.418E-04 | global batch size:   256 | lm loss: 3.303015E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.769 | TFLOPs: 32.38 |
0: [2023-03-17 06:25:00,269] [INFO] [logging.py:68:log_dist] [Rank 0] step=68000, skipped=0, lr=[0.00014160436454810027, 0.00014160436454810027, 0.00014160436454810027], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    68000/  173500 | consumed samples:     17408000 | consumed tokens:  35651584000 | elapsed time per iteration (s): 0.37 | learning rate: 1.416E-04 | global batch size:   256 | lm loss: 3.303881E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.977 | TFLOPs: 32.39 |
0: steps: 68000 loss: 3.2973 iter time (s): 0.367 samples/sec: 697.157
7:  iteration    68100/  173500 | consumed samples:     17433600 | consumed tokens:  35704012800 | elapsed time per iteration (s): 0.37 | learning rate: 1.415E-04 | global batch size:   256 | lm loss: 3.304727E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.462 | TFLOPs: 32.41 |
7:  iteration    68200/  173500 | consumed samples:     17459200 | consumed tokens:  35756441600 | elapsed time per iteration (s): 0.37 | learning rate: 1.413E-04 | global batch size:   256 | lm loss: 3.300490E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.822 | TFLOPs: 32.39 |
7:  iteration    68300/  173500 | consumed samples:     17484800 | consumed tokens:  35808870400 | elapsed time per iteration (s): 0.37 | learning rate: 1.411E-04 | global batch size:   256 | lm loss: 3.305536E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.241 | TFLOPs: 32.40 |
7:  iteration    68400/  173500 | consumed samples:     17510400 | consumed tokens:  35861299200 | elapsed time per iteration (s): 0.37 | learning rate: 1.410E-04 | global batch size:   256 | lm loss: 3.300256E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.078 | TFLOPs: 32.44 |
7:  iteration    68500/  173500 | consumed samples:     17536000 | consumed tokens:  35913728000 | elapsed time per iteration (s): 0.37 | learning rate: 1.408E-04 | global batch size:   256 | lm loss: 3.305309E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.978 | TFLOPs: 32.39 |
7:  iteration    68600/  173500 | consumed samples:     17561600 | consumed tokens:  35966156800 | elapsed time per iteration (s): 0.37 | learning rate: 1.407E-04 | global batch size:   256 | lm loss: 3.304293E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.313 | TFLOPs: 32.41 |
7:  iteration    68700/  173500 | consumed samples:     17587200 | consumed tokens:  36018585600 | elapsed time per iteration (s): 0.37 | learning rate: 1.405E-04 | global batch size:   256 | lm loss: 3.300862E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.302 | TFLOPs: 32.31 |
7:  iteration    68800/  173500 | consumed samples:     17612800 | consumed tokens:  36071014400 | elapsed time per iteration (s): 0.37 | learning rate: 1.404E-04 | global batch size:   256 | lm loss: 3.309284E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.316 | TFLOPs: 32.31 |
7:  iteration    68900/  173500 | consumed samples:     17638400 | consumed tokens:  36123443200 | elapsed time per iteration (s): 0.37 | learning rate: 1.402E-04 | global batch size:   256 | lm loss: 3.301981E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.219 | TFLOPs: 32.31 |
7:  iteration    69000/  173500 | consumed samples:     17664000 | consumed tokens:  36175872000 | elapsed time per iteration (s): 0.37 | learning rate: 1.401E-04 | global batch size:   256 | lm loss: 3.299660E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.708 | TFLOPs: 32.29 |
7:  iteration    69100/  173500 | consumed samples:     17689600 | consumed tokens:  36228300800 | elapsed time per iteration (s): 0.37 | learning rate: 1.399E-04 | global batch size:   256 | lm loss: 3.304025E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.794 | TFLOPs: 32.38 |
7:  iteration    69200/  173500 | consumed samples:     17715200 | consumed tokens:  36280729600 | elapsed time per iteration (s): 0.37 | learning rate: 1.397E-04 | global batch size:   256 | lm loss: 3.298297E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.187 | TFLOPs: 32.40 |
7:  iteration    69300/  173500 | consumed samples:     17740800 | consumed tokens:  36333158400 | elapsed time per iteration (s): 0.37 | learning rate: 1.396E-04 | global batch size:   256 | lm loss: 3.300083E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.596 | TFLOPs: 32.42 |
7:  iteration    69400/  173500 | consumed samples:     17766400 | consumed tokens:  36385587200 | elapsed time per iteration (s): 0.37 | learning rate: 1.394E-04 | global batch size:   256 | lm loss: 3.295975E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.601 | TFLOPs: 32.42 |
7:  iteration    69500/  173500 | consumed samples:     17792000 | consumed tokens:  36438016000 | elapsed time per iteration (s): 0.37 | learning rate: 1.393E-04 | global batch size:   256 | lm loss: 3.303292E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.373 | TFLOPs: 32.41 |
7:  iteration    69600/  173500 | consumed samples:     17817600 | consumed tokens:  36490444800 | elapsed time per iteration (s): 0.37 | learning rate: 1.391E-04 | global batch size:   256 | lm loss: 3.299023E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.704 | TFLOPs: 32.38 |
7:  iteration    69700/  173500 | consumed samples:     17843200 | consumed tokens:  36542873600 | elapsed time per iteration (s): 0.37 | learning rate: 1.390E-04 | global batch size:   256 | lm loss: 3.300209E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.511 | TFLOPs: 32.37 |
7:  iteration    69800/  173500 | consumed samples:     17868800 | consumed tokens:  36595302400 | elapsed time per iteration (s): 0.37 | learning rate: 1.388E-04 | global batch size:   256 | lm loss: 3.298258E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.928 | TFLOPs: 32.25 |
7:  iteration    69900/  173500 | consumed samples:     17894400 | consumed tokens:  36647731200 | elapsed time per iteration (s): 0.37 | learning rate: 1.387E-04 | global batch size:   256 | lm loss: 3.299525E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.492 | TFLOPs: 32.37 |
0: [2023-03-17 06:37:18,645] [INFO] [logging.py:68:log_dist] [Rank 0] step=70000, skipped=0, lr=[0.0001385013705497804, 0.0001385013705497804, 0.0001385013705497804], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    70000/  173500 | consumed samples:     17920000 | consumed tokens:  36700160000 | elapsed time per iteration (s): 0.37 | learning rate: 1.385E-04 | global batch size:   256 | lm loss: 3.300305E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.694 | TFLOPs: 32.24 |
0: steps: 70000 loss: 3.2480 iter time (s): 0.368 samples/sec: 696.349
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 70000 | lm loss value: 3.405876E+00 | lm loss PPL: 3.014069E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   70000 to checkpoints_146m91b400m
0: [2023-03-17 06:37:18,770] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step70000 is begin to save!
0: [2023-03-17 06:37:18,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_01-model_00-model_states.pt...
0: [2023-03-17 06:37:18,871] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_01-model_00-model_states.pt.
0: [2023-03-17 06:37:18,871] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_03-model_00-model_states.pt...
0: [2023-03-17 06:37:18,886] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_03-model_00-model_states.pt.
0: [2023-03-17 06:37:18,887] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_04-model_00-model_states.pt...
0: [2023-03-17 06:37:18,901] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_04-model_00-model_states.pt.
0: [2023-03-17 06:37:18,902] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_05-model_00-model_states.pt...
0: [2023-03-17 06:37:18,917] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_05-model_00-model_states.pt.
0: [2023-03-17 06:37:18,917] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_06-model_00-model_states.pt...
0: [2023-03-17 06:37:18,931] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_06-model_00-model_states.pt.
0: [2023-03-17 06:37:18,932] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_07-model_00-model_states.pt...
0: [2023-03-17 06:37:18,946] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_07-model_00-model_states.pt.
0: [2023-03-17 06:37:18,947] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_08-model_00-model_states.pt...
0: [2023-03-17 06:37:18,961] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_08-model_00-model_states.pt.
0: [2023-03-17 06:37:18,961] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_09-model_00-model_states.pt...
0: [2023-03-17 06:37:18,976] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_09-model_00-model_states.pt.
0: [2023-03-17 06:37:18,976] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_10-model_00-model_states.pt...
0: [2023-03-17 06:37:18,991] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_10-model_00-model_states.pt.
0: [2023-03-17 06:37:18,991] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_11-model_00-model_states.pt...
0: [2023-03-17 06:37:19,006] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_11-model_00-model_states.pt.
0: [2023-03-17 06:37:19,006] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_12-model_00-model_states.pt...
0: [2023-03-17 06:37:19,021] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_12-model_00-model_states.pt.
0: [2023-03-17 06:37:19,021] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_13-model_00-model_states.pt...
0: [2023-03-17 06:37:19,036] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_13-model_00-model_states.pt.
0: [2023-03-17 06:37:19,036] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_14-model_00-model_states.pt...
0: [2023-03-17 06:37:19,051] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_14-model_00-model_states.pt.
0: [2023-03-17 06:37:19,051] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_15-model_00-model_states.pt...
0: [2023-03-17 06:37:19,066] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_15-model_00-model_states.pt.
0: [2023-03-17 06:37:19,066] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_16-model_00-model_states.pt...
0: [2023-03-17 06:37:19,081] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_16-model_00-model_states.pt.
0: [2023-03-17 06:37:19,081] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_17-model_00-model_states.pt...
0: [2023-03-17 06:37:19,096] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_17-model_00-model_states.pt.
0: [2023-03-17 06:37:19,096] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/layer_19-model_00-model_states.pt...
0: [2023-03-17 06:37:19,097] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/layer_19-model_00-model_states.pt.
0: [2023-03-17 06:37:19,098] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step70000/mp_rank_00_model_states.pt
0: [2023-03-17 06:37:19,098] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/mp_rank_00_model_states.pt...
0: [2023-03-17 06:37:19,100] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/mp_rank_00_model_states.pt.
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
0: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 06:37:19,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
7: [2023-03-17 06:37:19,153] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,153] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,153] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,153] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,153] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,153] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,154] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,154] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,155] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,155] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,155] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,155] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,156] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,156] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,156] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,157] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,157] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,157] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,158] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,158] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
6: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 06:37:19,158] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,158] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,158] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
1: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 06:37:19,162] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
7: [2023-03-17 06:37:19,162] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,169] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
3: [2023-03-17 06:37:19,170] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 06:37:19,170] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 06:37:19,170] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,180] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0: [2023-03-17 06:37:19,181] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 06:37:19,181] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
5: [2023-03-17 06:37:19,184] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
4: [2023-03-17 06:37:19,192] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 06:37:19,192] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 06:37:19,192] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
2: [2023-03-17 06:37:19,196] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now!
0:   successfully saved checkpoint at iteration   70000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 433.14
7:  iteration    70100/  173500 | consumed samples:     17945600 | consumed tokens:  36752588800 | elapsed time per iteration (s): 0.38 | learning rate: 1.383E-04 | global batch size:   256 | lm loss: 3.300227E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.787 | TFLOPs: 31.68 |
7:  iteration    70200/  173500 | consumed samples:     17971200 | consumed tokens:  36805017600 | elapsed time per iteration (s): 0.37 | learning rate: 1.382E-04 | global batch size:   256 | lm loss: 3.298226E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.135 | TFLOPs: 32.07 |
7:  iteration    70300/  173500 | consumed samples:     17996800 | consumed tokens:  36857446400 | elapsed time per iteration (s): 0.37 | learning rate: 1.380E-04 | global batch size:   256 | lm loss: 3.300115E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.636 | TFLOPs: 32.24 |
7:  iteration    70400/  173500 | consumed samples:     18022400 | consumed tokens:  36909875200 | elapsed time per iteration (s): 0.37 | learning rate: 1.379E-04 | global batch size:   256 | lm loss: 3.302004E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.738 | TFLOPs: 32.29 |
7:  iteration    70500/  173500 | consumed samples:     18048000 | consumed tokens:  36962304000 | elapsed time per iteration (s): 0.37 | learning rate: 1.377E-04 | global batch size:   256 | lm loss: 3.301194E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.922 | TFLOPs: 32.25 |
7:  iteration    70600/  173500 | consumed samples:     18073600 | consumed tokens:  37014732800 | elapsed time per iteration (s): 0.38 | learning rate: 1.376E-04 | global batch size:   256 | lm loss: 3.296692E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.661 | TFLOPs: 31.77 |
7:  iteration    70700/  173500 | consumed samples:     18099200 | consumed tokens:  37067161600 | elapsed time per iteration (s): 0.37 | learning rate: 1.374E-04 | global batch size:   256 | lm loss: 3.299408E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.864 | TFLOPs: 32.25 |
7:  iteration    70800/  173500 | consumed samples:     18124800 | consumed tokens:  37119590400 | elapsed time per iteration (s): 0.37 | learning rate: 1.372E-04 | global batch size:   256 | lm loss: 3.303323E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.911 | TFLOPs: 32.39 |
7:  iteration    70900/  173500 | consumed samples:     18150400 | consumed tokens:  37172019200 | elapsed time per iteration (s): 0.37 | learning rate: 1.371E-04 | global batch size:   256 | lm loss: 3.292332E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.404 | TFLOPs: 32.41 |
7:  iteration    71000/  173500 | consumed samples:     18176000 | consumed tokens:  37224448000 | elapsed time per iteration (s): 0.37 | learning rate: 1.369E-04 | global batch size:   256 | lm loss: 3.293949E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.984 | TFLOPs: 32.39 |
7:  iteration    71100/  173500 | consumed samples:     18201600 | consumed tokens:  37276876800 | elapsed time per iteration (s): 0.37 | learning rate: 1.368E-04 | global batch size:   256 | lm loss: 3.301109E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.569 | TFLOPs: 32.28 |
7:  iteration    71200/  173500 | consumed samples:     18227200 | consumed tokens:  37329305600 | elapsed time per iteration (s): 0.37 | learning rate: 1.366E-04 | global batch size:   256 | lm loss: 3.298636E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.949 | TFLOPs: 32.34 |
7:  iteration    71300/  173500 | consumed samples:     18252800 | consumed tokens:  37381734400 | elapsed time per iteration (s): 0.37 | learning rate: 1.365E-04 | global batch size:   256 | lm loss: 3.296004E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.669 | TFLOPs: 32.33 |
7:  iteration    71400/  173500 | consumed samples:     18278400 | consumed tokens:  37434163200 | elapsed time per iteration (s): 0.37 | learning rate: 1.363E-04 | global batch size:   256 | lm loss: 3.298159E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.645 | TFLOPs: 32.42 |
7:  iteration    71500/  173500 | consumed samples:     18304000 | consumed tokens:  37486592000 | elapsed time per iteration (s): 0.37 | learning rate: 1.361E-04 | global batch size:   256 | lm loss: 3.296079E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.086 | TFLOPs: 32.30 |
7:  iteration    71600/  173500 | consumed samples:     18329600 | consumed tokens:  37539020800 | elapsed time per iteration (s): 0.37 | learning rate: 1.360E-04 | global batch size:   256 | lm loss: 3.295640E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.833 | TFLOPs: 32.39 |
7:  iteration    71700/  173500 | consumed samples:     18355200 | consumed tokens:  37591449600 | elapsed time per iteration (s): 0.37 | learning rate: 1.358E-04 | global batch size:   256 | lm loss: 3.296986E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.782 | TFLOPs: 32.29 |
7:  iteration    71800/  173500 | consumed samples:     18380800 | consumed tokens:  37643878400 | elapsed time per iteration (s): 0.37 | learning rate: 1.357E-04 | global batch size:   256 | lm loss: 3.300660E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.946 | TFLOPs: 32.30 |
7:  iteration    71900/  173500 | consumed samples:     18406400 | consumed tokens:  37696307200 | elapsed time per iteration (s): 0.37 | learning rate: 1.355E-04 | global batch size:   256 | lm loss: 3.293256E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.221 | TFLOPs: 32.31 |
0: [2023-03-17 06:49:39,708] [INFO] [logging.py:68:log_dist] [Rank 0] step=72000, skipped=0, lr=[0.0001353602432066091, 0.0001353602432066091, 0.0001353602432066091], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    72000/  173500 | consumed samples:     18432000 | consumed tokens:  37748736000 | elapsed time per iteration (s): 0.37 | learning rate: 1.354E-04 | global batch size:   256 | lm loss: 3.296673E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.509 | TFLOPs: 32.28 |
0: steps: 72000 loss: 3.2969 iter time (s): 0.368 samples/sec: 694.913
7:  iteration    72100/  173500 | consumed samples:     18457600 | consumed tokens:  37801164800 | elapsed time per iteration (s): 0.37 | learning rate: 1.352E-04 | global batch size:   256 | lm loss: 3.298240E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.669 | TFLOPs: 32.28 |
7:  iteration    72200/  173500 | consumed samples:     18483200 | consumed tokens:  37853593600 | elapsed time per iteration (s): 0.37 | learning rate: 1.350E-04 | global batch size:   256 | lm loss: 3.300305E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.394 | TFLOPs: 32.27 |
7:  iteration    72300/  173500 | consumed samples:     18508800 | consumed tokens:  37906022400 | elapsed time per iteration (s): 0.37 | learning rate: 1.349E-04 | global batch size:   256 | lm loss: 3.298539E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.379 | TFLOPs: 32.04 |
7:  iteration    72400/  173500 | consumed samples:     18534400 | consumed tokens:  37958451200 | elapsed time per iteration (s): 0.38 | learning rate: 1.347E-04 | global batch size:   256 | lm loss: 3.292917E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.350 | TFLOPs: 31.85 |
7:  iteration    72500/  173500 | consumed samples:     18560000 | consumed tokens:  38010880000 | elapsed time per iteration (s): 0.37 | learning rate: 1.346E-04 | global batch size:   256 | lm loss: 3.297978E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.898 | TFLOPs: 32.16 |
7:  iteration    72600/  173500 | consumed samples:     18585600 | consumed tokens:  38063308800 | elapsed time per iteration (s): 0.37 | learning rate: 1.344E-04 | global batch size:   256 | lm loss: 3.293317E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.586 | TFLOPs: 32.28 |
7:  iteration    72700/  173500 | consumed samples:     18611200 | consumed tokens:  38115737600 | elapsed time per iteration (s): 0.37 | learning rate: 1.343E-04 | global batch size:   256 | lm loss: 3.297573E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.185 | TFLOPs: 32.12 |
7:  iteration    72800/  173500 | consumed samples:     18636800 | consumed tokens:  38168166400 | elapsed time per iteration (s): 0.37 | learning rate: 1.341E-04 | global batch size:   256 | lm loss: 3.294077E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.613 | TFLOPs: 32.24 |
7:  iteration    72900/  173500 | consumed samples:     18662400 | consumed tokens:  38220595200 | elapsed time per iteration (s): 0.37 | learning rate: 1.339E-04 | global batch size:   256 | lm loss: 3.292202E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.093 | TFLOPs: 32.12 |
7:  iteration    73000/  173500 | consumed samples:     18688000 | consumed tokens:  38273024000 | elapsed time per iteration (s): 0.37 | learning rate: 1.338E-04 | global batch size:   256 | lm loss: 3.297819E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.097 | TFLOPs: 32.26 |
7:  iteration    73100/  173500 | consumed samples:     18713600 | consumed tokens:  38325452800 | elapsed time per iteration (s): 0.37 | learning rate: 1.336E-04 | global batch size:   256 | lm loss: 3.295537E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.204 | TFLOPs: 32.17 |
7:  iteration    73200/  173500 | consumed samples:     18739200 | consumed tokens:  38377881600 | elapsed time per iteration (s): 0.37 | learning rate: 1.335E-04 | global batch size:   256 | lm loss: 3.299612E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.894 | TFLOPs: 32.39 |
7:  iteration    73300/  173500 | consumed samples:     18764800 | consumed tokens:  38430310400 | elapsed time per iteration (s): 0.37 | learning rate: 1.333E-04 | global batch size:   256 | lm loss: 3.296785E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.696 | TFLOPs: 32.38 |
7:  iteration    73400/  173500 | consumed samples:     18790400 | consumed tokens:  38482739200 | elapsed time per iteration (s): 0.37 | learning rate: 1.331E-04 | global batch size:   256 | lm loss: 3.295899E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.340 | TFLOPs: 32.18 |
7:  iteration    73500/  173500 | consumed samples:     18816000 | consumed tokens:  38535168000 | elapsed time per iteration (s): 0.37 | learning rate: 1.330E-04 | global batch size:   256 | lm loss: 3.290430E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.374 | TFLOPs: 32.13 |
7:  iteration    73600/  173500 | consumed samples:     18841600 | consumed tokens:  38587596800 | elapsed time per iteration (s): 0.37 | learning rate: 1.328E-04 | global batch size:   256 | lm loss: 3.295294E+00 | grad norm: 0.204 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.910 | TFLOPs: 31.97 |
7:  iteration    73700/  173500 | consumed samples:     18867200 | consumed tokens:  38640025600 | elapsed time per iteration (s): 0.37 | learning rate: 1.327E-04 | global batch size:   256 | lm loss: 3.298562E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.690 | TFLOPs: 32.15 |
7:  iteration    73800/  173500 | consumed samples:     18892800 | consumed tokens:  38692454400 | elapsed time per iteration (s): 0.37 | learning rate: 1.325E-04 | global batch size:   256 | lm loss: 3.296371E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.257 | TFLOPs: 32.22 |
7:  iteration    73900/  173500 | consumed samples:     18918400 | consumed tokens:  38744883200 | elapsed time per iteration (s): 0.37 | learning rate: 1.323E-04 | global batch size:   256 | lm loss: 3.299644E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.908 | TFLOPs: 32.25 |
0: [2023-03-17 07:02:02,082] [INFO] [logging.py:68:log_dist] [Rank 0] step=74000, skipped=0, lr=[0.0001321851851828754, 0.0001321851851828754, 0.0001321851851828754], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    74000/  173500 | consumed samples:     18944000 | consumed tokens:  38797312000 | elapsed time per iteration (s): 0.37 | learning rate: 1.322E-04 | global batch size:   256 | lm loss: 3.295780E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.526 | TFLOPs: 32.42 |
0: steps: 74000 loss: 3.2878 iter time (s): 0.369 samples/sec: 693.629
7:  iteration    74100/  173500 | consumed samples:     18969600 | consumed tokens:  38849740800 | elapsed time per iteration (s): 0.37 | learning rate: 1.320E-04 | global batch size:   256 | lm loss: 3.293750E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.393 | TFLOPs: 32.37 |
7:  iteration    74200/  173500 | consumed samples:     18995200 | consumed tokens:  38902169600 | elapsed time per iteration (s): 0.37 | learning rate: 1.319E-04 | global batch size:   256 | lm loss: 3.289422E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.063 | TFLOPs: 32.35 |
7:  iteration    74300/  173500 | consumed samples:     19020800 | consumed tokens:  38954598400 | elapsed time per iteration (s): 0.37 | learning rate: 1.317E-04 | global batch size:   256 | lm loss: 3.293338E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.618 | TFLOPs: 32.42 |
7:  iteration    74400/  173500 | consumed samples:     19046400 | consumed tokens:  39007027200 | elapsed time per iteration (s): 0.37 | learning rate: 1.315E-04 | global batch size:   256 | lm loss: 3.290053E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.395 | TFLOPs: 32.41 |
7:  iteration    74500/  173500 | consumed samples:     19072000 | consumed tokens:  39059456000 | elapsed time per iteration (s): 0.37 | learning rate: 1.314E-04 | global batch size:   256 | lm loss: 3.293278E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.516 | TFLOPs: 32.42 |
7:  iteration    74600/  173500 | consumed samples:     19097600 | consumed tokens:  39111884800 | elapsed time per iteration (s): 0.37 | learning rate: 1.312E-04 | global batch size:   256 | lm loss: 3.291194E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.737 | TFLOPs: 32.43 |
7:  iteration    74700/  173500 | consumed samples:     19123200 | consumed tokens:  39164313600 | elapsed time per iteration (s): 0.37 | learning rate: 1.311E-04 | global batch size:   256 | lm loss: 3.294375E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.315 | TFLOPs: 32.41 |
7:  iteration    74800/  173500 | consumed samples:     19148800 | consumed tokens:  39216742400 | elapsed time per iteration (s): 0.37 | learning rate: 1.309E-04 | global batch size:   256 | lm loss: 3.296916E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.550 | TFLOPs: 32.42 |
7:  iteration    74900/  173500 | consumed samples:     19174400 | consumed tokens:  39269171200 | elapsed time per iteration (s): 0.37 | learning rate: 1.307E-04 | global batch size:   256 | lm loss: 3.292783E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.859 | TFLOPs: 32.43 |
7:  iteration    75000/  173500 | consumed samples:     19200000 | consumed tokens:  39321600000 | elapsed time per iteration (s): 0.37 | learning rate: 1.306E-04 | global batch size:   256 | lm loss: 3.293409E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.137 | TFLOPs: 32.45 |
7:  iteration    75100/  173500 | consumed samples:     19225600 | consumed tokens:  39374028800 | elapsed time per iteration (s): 0.37 | learning rate: 1.304E-04 | global batch size:   256 | lm loss: 3.293113E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.203 | TFLOPs: 32.45 |
7:  iteration    75200/  173500 | consumed samples:     19251200 | consumed tokens:  39426457600 | elapsed time per iteration (s): 0.37 | learning rate: 1.303E-04 | global batch size:   256 | lm loss: 3.297559E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.364 | TFLOPs: 32.46 |
7:  iteration    75300/  173500 | consumed samples:     19276800 | consumed tokens:  39478886400 | elapsed time per iteration (s): 0.37 | learning rate: 1.301E-04 | global batch size:   256 | lm loss: 3.287543E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.512 | TFLOPs: 32.42 |
7:  iteration    75400/  173500 | consumed samples:     19302400 | consumed tokens:  39531315200 | elapsed time per iteration (s): 0.37 | learning rate: 1.299E-04 | global batch size:   256 | lm loss: 3.290447E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.086 | TFLOPs: 32.40 |
7:  iteration    75500/  173500 | consumed samples:     19328000 | consumed tokens:  39583744000 | elapsed time per iteration (s): 0.37 | learning rate: 1.298E-04 | global batch size:   256 | lm loss: 3.291186E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.029 | TFLOPs: 32.44 |
7:  iteration    75600/  173500 | consumed samples:     19353600 | consumed tokens:  39636172800 | elapsed time per iteration (s): 0.37 | learning rate: 1.296E-04 | global batch size:   256 | lm loss: 3.289949E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.791 | TFLOPs: 32.43 |
7:  iteration    75700/  173500 | consumed samples:     19379200 | consumed tokens:  39688601600 | elapsed time per iteration (s): 0.37 | learning rate: 1.295E-04 | global batch size:   256 | lm loss: 3.295755E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.667 | TFLOPs: 32.42 |
7:  iteration    75800/  173500 | consumed samples:     19404800 | consumed tokens:  39741030400 | elapsed time per iteration (s): 0.37 | learning rate: 1.293E-04 | global batch size:   256 | lm loss: 3.295343E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.937 | TFLOPs: 32.39 |
7:  iteration    75900/  173500 | consumed samples:     19430400 | consumed tokens:  39793459200 | elapsed time per iteration (s): 0.37 | learning rate: 1.291E-04 | global batch size:   256 | lm loss: 3.289371E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.465 | TFLOPs: 32.42 |
0: [2023-03-17 07:14:19,294] [INFO] [logging.py:68:log_dist] [Rank 0] step=76000, skipped=0, lr=[0.0001289804445403464, 0.0001289804445403464, 0.0001289804445403464], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    76000/  173500 | consumed samples:     19456000 | consumed tokens:  39845888000 | elapsed time per iteration (s): 0.37 | learning rate: 1.290E-04 | global batch size:   256 | lm loss: 3.288922E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.543 | TFLOPs: 32.42 |
0: steps: 76000 loss: 3.2746 iter time (s): 0.367 samples/sec: 698.297
7:  iteration    76100/  173500 | consumed samples:     19481600 | consumed tokens:  39898316800 | elapsed time per iteration (s): 0.37 | learning rate: 1.288E-04 | global batch size:   256 | lm loss: 3.286216E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.022 | TFLOPs: 32.30 |
7:  iteration    76200/  173500 | consumed samples:     19507200 | consumed tokens:  39950745600 | elapsed time per iteration (s): 0.37 | learning rate: 1.287E-04 | global batch size:   256 | lm loss: 3.290770E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.764 | TFLOPs: 32.43 |
7:  iteration    76300/  173500 | consumed samples:     19532800 | consumed tokens:  40003174400 | elapsed time per iteration (s): 0.37 | learning rate: 1.285E-04 | global batch size:   256 | lm loss: 3.292521E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.573 | TFLOPs: 32.42 |
7:  iteration    76400/  173500 | consumed samples:     19558400 | consumed tokens:  40055603200 | elapsed time per iteration (s): 0.37 | learning rate: 1.283E-04 | global batch size:   256 | lm loss: 3.286554E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.022 | TFLOPs: 32.39 |
7:  iteration    76500/  173500 | consumed samples:     19584000 | consumed tokens:  40108032000 | elapsed time per iteration (s): 0.37 | learning rate: 1.282E-04 | global batch size:   256 | lm loss: 3.292374E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.928 | TFLOPs: 32.39 |
7:  iteration    76600/  173500 | consumed samples:     19609600 | consumed tokens:  40160460800 | elapsed time per iteration (s): 0.37 | learning rate: 1.280E-04 | global batch size:   256 | lm loss: 3.287778E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.217 | TFLOPs: 32.40 |
7:  iteration    76700/  173500 | consumed samples:     19635200 | consumed tokens:  40212889600 | elapsed time per iteration (s): 0.37 | learning rate: 1.279E-04 | global batch size:   256 | lm loss: 3.291166E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.103 | TFLOPs: 32.40 |
7:  iteration    76800/  173500 | consumed samples:     19660800 | consumed tokens:  40265318400 | elapsed time per iteration (s): 0.37 | learning rate: 1.277E-04 | global batch size:   256 | lm loss: 3.291978E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.550 | TFLOPs: 32.19 |
7:  iteration    76900/  173500 | consumed samples:     19686400 | consumed tokens:  40317747200 | elapsed time per iteration (s): 0.37 | learning rate: 1.275E-04 | global batch size:   256 | lm loss: 3.289026E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.698 | TFLOPs: 32.43 |
7:  iteration    77000/  173500 | consumed samples:     19712000 | consumed tokens:  40370176000 | elapsed time per iteration (s): 0.37 | learning rate: 1.274E-04 | global batch size:   256 | lm loss: 3.289977E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.144 | TFLOPs: 32.40 |
7:  iteration    77100/  173500 | consumed samples:     19737600 | consumed tokens:  40422604800 | elapsed time per iteration (s): 0.37 | learning rate: 1.272E-04 | global batch size:   256 | lm loss: 3.290276E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.634 | TFLOPs: 32.42 |
7:  iteration    77200/  173500 | consumed samples:     19763200 | consumed tokens:  40475033600 | elapsed time per iteration (s): 0.37 | learning rate: 1.270E-04 | global batch size:   256 | lm loss: 3.289669E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.507 | TFLOPs: 32.42 |
7:  iteration    77300/  173500 | consumed samples:     19788800 | consumed tokens:  40527462400 | elapsed time per iteration (s): 0.37 | learning rate: 1.269E-04 | global batch size:   256 | lm loss: 3.291171E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.609 | TFLOPs: 32.42 |
7:  iteration    77400/  173500 | consumed samples:     19814400 | consumed tokens:  40579891200 | elapsed time per iteration (s): 0.37 | learning rate: 1.267E-04 | global batch size:   256 | lm loss: 3.290520E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.595 | TFLOPs: 32.42 |
7:  iteration    77500/  173500 | consumed samples:     19840000 | consumed tokens:  40632320000 | elapsed time per iteration (s): 0.37 | learning rate: 1.266E-04 | global batch size:   256 | lm loss: 3.290941E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.019 | TFLOPs: 32.44 |
7:  iteration    77600/  173500 | consumed samples:     19865600 | consumed tokens:  40684748800 | elapsed time per iteration (s): 0.37 | learning rate: 1.264E-04 | global batch size:   256 | lm loss: 3.289915E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.155 | TFLOPs: 32.45 |
7:  iteration    77700/  173500 | consumed samples:     19891200 | consumed tokens:  40737177600 | elapsed time per iteration (s): 0.37 | learning rate: 1.262E-04 | global batch size:   256 | lm loss: 3.284506E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.300 | TFLOPs: 32.36 |
7:  iteration    77800/  173500 | consumed samples:     19916800 | consumed tokens:  40789606400 | elapsed time per iteration (s): 0.37 | learning rate: 1.261E-04 | global batch size:   256 | lm loss: 3.286936E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.399 | TFLOPs: 32.46 |
7:  iteration    77900/  173500 | consumed samples:     19942400 | consumed tokens:  40842035200 | elapsed time per iteration (s): 0.37 | learning rate: 1.259E-04 | global batch size:   256 | lm loss: 3.288060E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.254 | TFLOPs: 32.45 |
0: [2023-03-17 07:26:36,873] [INFO] [logging.py:68:log_dist] [Rank 0] step=78000, skipped=0, lr=[0.00012575030905458257, 0.00012575030905458257, 0.00012575030905458257], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    78000/  173500 | consumed samples:     19968000 | consumed tokens:  40894464000 | elapsed time per iteration (s): 0.37 | learning rate: 1.258E-04 | global batch size:   256 | lm loss: 3.284525E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.832 | TFLOPs: 32.43 |
0: steps: 78000 loss: 3.2983 iter time (s): 0.367 samples/sec: 697.917
7:  iteration    78100/  173500 | consumed samples:     19993600 | consumed tokens:  40946892800 | elapsed time per iteration (s): 0.37 | learning rate: 1.256E-04 | global batch size:   256 | lm loss: 3.287291E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.666 | TFLOPs: 32.42 |
7:  iteration    78200/  173500 | consumed samples:     20019200 | consumed tokens:  40999321600 | elapsed time per iteration (s): 0.37 | learning rate: 1.254E-04 | global batch size:   256 | lm loss: 3.287923E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.949 | TFLOPs: 32.34 |
7:  iteration    78300/  173500 | consumed samples:     20044800 | consumed tokens:  41051750400 | elapsed time per iteration (s): 0.37 | learning rate: 1.253E-04 | global batch size:   256 | lm loss: 3.289804E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.895 | TFLOPs: 32.16 |
7:  iteration    78400/  173500 | consumed samples:     20070400 | consumed tokens:  41104179200 | elapsed time per iteration (s): 0.37 | learning rate: 1.251E-04 | global batch size:   256 | lm loss: 3.292086E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.992 | TFLOPs: 32.35 |
7:  iteration    78500/  173500 | consumed samples:     20096000 | consumed tokens:  41156608000 | elapsed time per iteration (s): 0.37 | learning rate: 1.249E-04 | global batch size:   256 | lm loss: 3.285729E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.853 | TFLOPs: 32.43 |
7:  iteration    78600/  173500 | consumed samples:     20121600 | consumed tokens:  41209036800 | elapsed time per iteration (s): 0.37 | learning rate: 1.248E-04 | global batch size:   256 | lm loss: 3.289775E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.248 | TFLOPs: 32.40 |
7:  iteration    78700/  173500 | consumed samples:     20147200 | consumed tokens:  41261465600 | elapsed time per iteration (s): 0.37 | learning rate: 1.246E-04 | global batch size:   256 | lm loss: 3.289619E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.115 | TFLOPs: 32.40 |
7:  iteration    78800/  173500 | consumed samples:     20172800 | consumed tokens:  41313894400 | elapsed time per iteration (s): 0.37 | learning rate: 1.245E-04 | global batch size:   256 | lm loss: 3.286403E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.168 | TFLOPs: 32.40 |
7:  iteration    78900/  173500 | consumed samples:     20198400 | consumed tokens:  41366323200 | elapsed time per iteration (s): 0.37 | learning rate: 1.243E-04 | global batch size:   256 | lm loss: 3.286686E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.475 | TFLOPs: 32.32 |
7:  iteration    79000/  173500 | consumed samples:     20224000 | consumed tokens:  41418752000 | elapsed time per iteration (s): 0.37 | learning rate: 1.241E-04 | global batch size:   256 | lm loss: 3.285245E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.991 | TFLOPs: 32.39 |
7:  iteration    79100/  173500 | consumed samples:     20249600 | consumed tokens:  41471180800 | elapsed time per iteration (s): 0.37 | learning rate: 1.240E-04 | global batch size:   256 | lm loss: 3.294496E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.055 | TFLOPs: 32.40 |
7:  iteration    79200/  173500 | consumed samples:     20275200 | consumed tokens:  41523609600 | elapsed time per iteration (s): 0.37 | learning rate: 1.238E-04 | global batch size:   256 | lm loss: 3.287019E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.578 | TFLOPs: 32.37 |
7:  iteration    79300/  173500 | consumed samples:     20300800 | consumed tokens:  41576038400 | elapsed time per iteration (s): 0.37 | learning rate: 1.236E-04 | global batch size:   256 | lm loss: 3.291662E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.887 | TFLOPs: 32.39 |
7:  iteration    79400/  173500 | consumed samples:     20326400 | consumed tokens:  41628467200 | elapsed time per iteration (s): 0.37 | learning rate: 1.235E-04 | global batch size:   256 | lm loss: 3.290418E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.551 | TFLOPs: 32.42 |
7:  iteration    79500/  173500 | consumed samples:     20352000 | consumed tokens:  41680896000 | elapsed time per iteration (s): 0.37 | learning rate: 1.233E-04 | global batch size:   256 | lm loss: 3.281866E+00 | grad norm: 0.238 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.030 | TFLOPs: 32.39 |
7:  iteration    79600/  173500 | consumed samples:     20377600 | consumed tokens:  41733324800 | elapsed time per iteration (s): 0.37 | learning rate: 1.232E-04 | global batch size:   256 | lm loss: 3.286853E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.386 | TFLOPs: 32.41 |
7:  iteration    79700/  173500 | consumed samples:     20403200 | consumed tokens:  41785753600 | elapsed time per iteration (s): 0.37 | learning rate: 1.230E-04 | global batch size:   256 | lm loss: 3.290515E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.440 | TFLOPs: 32.41 |
7:  iteration    79800/  173500 | consumed samples:     20428800 | consumed tokens:  41838182400 | elapsed time per iteration (s): 0.37 | learning rate: 1.228E-04 | global batch size:   256 | lm loss: 3.281570E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.461 | TFLOPs: 32.41 |
7:  iteration    79900/  173500 | consumed samples:     20454400 | consumed tokens:  41890611200 | elapsed time per iteration (s): 0.37 | learning rate: 1.227E-04 | global batch size:   256 | lm loss: 3.283387E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.805 | TFLOPs: 32.43 |
0: [2023-03-17 07:38:54,850] [INFO] [logging.py:68:log_dist] [Rank 0] step=80000, skipped=0, lr=[0.00012249910047811783, 0.00012249910047811783, 0.00012249910047811783], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    80000/  173500 | consumed samples:     20480000 | consumed tokens:  41943040000 | elapsed time per iteration (s): 0.37 | learning rate: 1.225E-04 | global batch size:   256 | lm loss: 3.285059E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.271 | TFLOPs: 32.41 |
0: steps: 80000 loss: 3.2963 iter time (s): 0.367 samples/sec: 697.282
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 80000 | lm loss value: 3.394659E+00 | lm loss PPL: 2.980450E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   80000 to checkpoints_146m91b400m
0: [2023-03-17 07:38:54,977] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step80000 is begin to save!
0: [2023-03-17 07:38:54,981] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_01-model_00-model_states.pt...
0: [2023-03-17 07:38:55,082] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_01-model_00-model_states.pt.
0: [2023-03-17 07:38:55,083] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_03-model_00-model_states.pt...
0: [2023-03-17 07:38:55,100] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_03-model_00-model_states.pt.
0: [2023-03-17 07:38:55,101] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_04-model_00-model_states.pt...
0: [2023-03-17 07:38:55,115] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_04-model_00-model_states.pt.
0: [2023-03-17 07:38:55,116] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_05-model_00-model_states.pt...
0: [2023-03-17 07:38:55,130] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_05-model_00-model_states.pt.
0: [2023-03-17 07:38:55,131] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_06-model_00-model_states.pt...
0: [2023-03-17 07:38:55,145] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_06-model_00-model_states.pt.
0: [2023-03-17 07:38:55,145] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_07-model_00-model_states.pt...
0: [2023-03-17 07:38:55,160] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_07-model_00-model_states.pt.
0: [2023-03-17 07:38:55,160] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_08-model_00-model_states.pt...
0: [2023-03-17 07:38:55,175] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_08-model_00-model_states.pt.
0: [2023-03-17 07:38:55,175] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_09-model_00-model_states.pt...
0: [2023-03-17 07:38:55,191] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_09-model_00-model_states.pt.
0: [2023-03-17 07:38:55,191] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_10-model_00-model_states.pt...
0: [2023-03-17 07:38:55,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_10-model_00-model_states.pt.
0: [2023-03-17 07:38:55,206] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_11-model_00-model_states.pt...
0: [2023-03-17 07:38:55,221] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_11-model_00-model_states.pt.
0: [2023-03-17 07:38:55,221] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_12-model_00-model_states.pt...
0: [2023-03-17 07:38:55,236] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_12-model_00-model_states.pt.
0: [2023-03-17 07:38:55,236] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_13-model_00-model_states.pt...
0: [2023-03-17 07:38:55,251] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_13-model_00-model_states.pt.
0: [2023-03-17 07:38:55,251] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_14-model_00-model_states.pt...
0: [2023-03-17 07:38:55,266] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_14-model_00-model_states.pt.
0: [2023-03-17 07:38:55,266] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_15-model_00-model_states.pt...
0: [2023-03-17 07:38:55,281] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_15-model_00-model_states.pt.
0: [2023-03-17 07:38:55,281] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_16-model_00-model_states.pt...
0: [2023-03-17 07:38:55,296] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_16-model_00-model_states.pt.
0: [2023-03-17 07:38:55,296] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_17-model_00-model_states.pt...
0: [2023-03-17 07:38:55,311] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_17-model_00-model_states.pt.
0: [2023-03-17 07:38:55,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/layer_19-model_00-model_states.pt...
0: [2023-03-17 07:38:55,312] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/layer_19-model_00-model_states.pt.
0: [2023-03-17 07:38:55,313] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step80000/mp_rank_00_model_states.pt
0: [2023-03-17 07:38:55,313] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/mp_rank_00_model_states.pt...
0: [2023-03-17 07:38:55,317] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/mp_rank_00_model_states.pt.
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
2: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
5: [2023-03-17 07:38:55,334] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
0: [2023-03-17 07:38:55,367] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,370] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,370] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,371] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,373] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,373] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,373] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,373] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,373] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,374] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,374] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,374] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,374] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,375] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,375] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,375] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
3: [2023-03-17 07:38:55,377] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 07:38:55,377] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 07:38:55,377] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,379] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,379] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,379] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,379] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,379] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,379] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,380] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,380] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,380] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,381] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,381] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,381] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,381] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,381] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,382] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,382] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 07:38:55,382] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,382] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 07:38:55,382] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
4: [2023-03-17 07:38:55,382] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0: [2023-03-17 07:38:55,399] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 07:38:55,399] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
5: [2023-03-17 07:38:55,406] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,409] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
7: [2023-03-17 07:38:55,410] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 07:38:55,410] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 07:38:55,410] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,411] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,411] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,411] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,412] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,412] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,412] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,412] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,412] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,412] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,413] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,413] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,413] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,413] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
1: [2023-03-17 07:38:55,413] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 07:38:55,413] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,422] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,423] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
2: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,424] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
6: [2023-03-17 07:38:55,428] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 07:38:55,428] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step80000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 07:38:55,428] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step80000 is ready now!
0:   successfully saved checkpoint at iteration   80000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 459.44
7:  iteration    80100/  173500 | consumed samples:     20505600 | consumed tokens:  41995468800 | elapsed time per iteration (s): 0.37 | learning rate: 1.223E-04 | global batch size:   256 | lm loss: 3.287162E+00 | grad norm: 0.201 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.176 | TFLOPs: 31.93 |
7:  iteration    80200/  173500 | consumed samples:     20531200 | consumed tokens:  42047897600 | elapsed time per iteration (s): 0.37 | learning rate: 1.222E-04 | global batch size:   256 | lm loss: 3.285111E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.935 | TFLOPs: 32.39 |
7:  iteration    80300/  173500 | consumed samples:     20556800 | consumed tokens:  42100326400 | elapsed time per iteration (s): 0.37 | learning rate: 1.220E-04 | global batch size:   256 | lm loss: 3.286188E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.834 | TFLOPs: 32.15 |
7:  iteration    80400/  173500 | consumed samples:     20582400 | consumed tokens:  42152755200 | elapsed time per iteration (s): 0.37 | learning rate: 1.218E-04 | global batch size:   256 | lm loss: 3.280395E+00 | grad norm: 0.198 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.752 | TFLOPs: 32.10 |
7:  iteration    80500/  173500 | consumed samples:     20608000 | consumed tokens:  42205184000 | elapsed time per iteration (s): 0.37 | learning rate: 1.217E-04 | global batch size:   256 | lm loss: 3.288355E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.529 | TFLOPs: 32.18 |
7:  iteration    80600/  173500 | consumed samples:     20633600 | consumed tokens:  42257612800 | elapsed time per iteration (s): 0.37 | learning rate: 1.215E-04 | global batch size:   256 | lm loss: 3.283978E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.804 | TFLOPs: 32.01 |
7:  iteration    80700/  173500 | consumed samples:     20659200 | consumed tokens:  42310041600 | elapsed time per iteration (s): 0.38 | learning rate: 1.214E-04 | global batch size:   256 | lm loss: 3.282912E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.486 | TFLOPs: 31.76 |
7:  iteration    80800/  173500 | consumed samples:     20684800 | consumed tokens:  42362470400 | elapsed time per iteration (s): 0.38 | learning rate: 1.212E-04 | global batch size:   256 | lm loss: 3.281368E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.522 | TFLOPs: 31.86 |
7:  iteration    80900/  173500 | consumed samples:     20710400 | consumed tokens:  42414899200 | elapsed time per iteration (s): 0.38 | learning rate: 1.210E-04 | global batch size:   256 | lm loss: 3.287358E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.998 | TFLOPs: 31.74 |
7:  iteration    81000/  173500 | consumed samples:     20736000 | consumed tokens:  42467328000 | elapsed time per iteration (s): 0.38 | learning rate: 1.209E-04 | global batch size:   256 | lm loss: 3.286899E+00 | grad norm: 0.196 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.859 | TFLOPs: 31.45 |
7:  iteration    81100/  173500 | consumed samples:     20761600 | consumed tokens:  42519756800 | elapsed time per iteration (s): 0.38 | learning rate: 1.207E-04 | global batch size:   256 | lm loss: 3.283528E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.417 | TFLOPs: 31.67 |
7:  iteration    81200/  173500 | consumed samples:     20787200 | consumed tokens:  42572185600 | elapsed time per iteration (s): 0.38 | learning rate: 1.205E-04 | global batch size:   256 | lm loss: 3.284479E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 671.291 | TFLOPs: 31.33 |
7:  iteration    81300/  173500 | consumed samples:     20812800 | consumed tokens:  42624614400 | elapsed time per iteration (s): 0.38 | learning rate: 1.204E-04 | global batch size:   256 | lm loss: 3.284256E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.814 | TFLOPs: 31.73 |
7:  iteration    81400/  173500 | consumed samples:     20838400 | consumed tokens:  42677043200 | elapsed time per iteration (s): 0.38 | learning rate: 1.202E-04 | global batch size:   256 | lm loss: 3.280571E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 675.934 | TFLOPs: 31.55 |
7:  iteration    81500/  173500 | consumed samples:     20864000 | consumed tokens:  42729472000 | elapsed time per iteration (s): 0.37 | learning rate: 1.200E-04 | global batch size:   256 | lm loss: 3.282179E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.826 | TFLOPs: 31.92 |
7:  iteration    81600/  173500 | consumed samples:     20889600 | consumed tokens:  42781900800 | elapsed time per iteration (s): 0.38 | learning rate: 1.199E-04 | global batch size:   256 | lm loss: 3.281188E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.456 | TFLOPs: 31.81 |
7:  iteration    81700/  173500 | consumed samples:     20915200 | consumed tokens:  42834329600 | elapsed time per iteration (s): 0.37 | learning rate: 1.197E-04 | global batch size:   256 | lm loss: 3.282370E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.187 | TFLOPs: 32.17 |
7:  iteration    81800/  173500 | consumed samples:     20940800 | consumed tokens:  42886758400 | elapsed time per iteration (s): 0.37 | learning rate: 1.196E-04 | global batch size:   256 | lm loss: 3.283431E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.760 | TFLOPs: 32.29 |
7:  iteration    81900/  173500 | consumed samples:     20966400 | consumed tokens:  42939187200 | elapsed time per iteration (s): 0.37 | learning rate: 1.194E-04 | global batch size:   256 | lm loss: 3.284086E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.990 | TFLOPs: 32.25 |
0: [2023-03-17 07:51:23,365] [INFO] [logging.py:68:log_dist] [Rank 0] step=82000, skipped=0, lr=[0.00011923116875818059, 0.00011923116875818059, 0.00011923116875818059], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    82000/  173500 | consumed samples:     20992000 | consumed tokens:  42991616000 | elapsed time per iteration (s): 0.37 | learning rate: 1.192E-04 | global batch size:   256 | lm loss: 3.279405E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.995 | TFLOPs: 32.30 |
0: steps: 82000 loss: 3.2876 iter time (s): 0.372 samples/sec: 687.345
7:  iteration    82100/  173500 | consumed samples:     21017600 | consumed tokens:  43044044800 | elapsed time per iteration (s): 0.37 | learning rate: 1.191E-04 | global batch size:   256 | lm loss: 3.286348E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.454 | TFLOPs: 32.09 |
7:  iteration    82200/  173500 | consumed samples:     21043200 | consumed tokens:  43096473600 | elapsed time per iteration (s): 0.37 | learning rate: 1.189E-04 | global batch size:   256 | lm loss: 3.281549E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.467 | TFLOPs: 32.23 |
7:  iteration    82300/  173500 | consumed samples:     21068800 | consumed tokens:  43148902400 | elapsed time per iteration (s): 0.38 | learning rate: 1.187E-04 | global batch size:   256 | lm loss: 3.281498E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.858 | TFLOPs: 31.73 |
7:  iteration    82400/  173500 | consumed samples:     21094400 | consumed tokens:  43201331200 | elapsed time per iteration (s): 0.37 | learning rate: 1.186E-04 | global batch size:   256 | lm loss: 3.283478E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.970 | TFLOPs: 32.11 |
7:  iteration    82500/  173500 | consumed samples:     21120000 | consumed tokens:  43253760000 | elapsed time per iteration (s): 0.37 | learning rate: 1.184E-04 | global batch size:   256 | lm loss: 3.283779E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.222 | TFLOPs: 32.26 |
7:  iteration    82600/  173500 | consumed samples:     21145600 | consumed tokens:  43306188800 | elapsed time per iteration (s): 0.37 | learning rate: 1.182E-04 | global batch size:   256 | lm loss: 3.283527E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.763 | TFLOPs: 32.20 |
7:  iteration    82700/  173500 | consumed samples:     21171200 | consumed tokens:  43358617600 | elapsed time per iteration (s): 0.37 | learning rate: 1.181E-04 | global batch size:   256 | lm loss: 3.279970E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.093 | TFLOPs: 32.21 |
7:  iteration    82800/  173500 | consumed samples:     21196800 | consumed tokens:  43411046400 | elapsed time per iteration (s): 0.37 | learning rate: 1.179E-04 | global batch size:   256 | lm loss: 3.286165E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.871 | TFLOPs: 32.25 |
7:  iteration    82900/  173500 | consumed samples:     21222400 | consumed tokens:  43463475200 | elapsed time per iteration (s): 0.37 | learning rate: 1.178E-04 | global batch size:   256 | lm loss: 3.282302E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.886 | TFLOPs: 32.25 |
7:  iteration    83000/  173500 | consumed samples:     21248000 | consumed tokens:  43515904000 | elapsed time per iteration (s): 0.37 | learning rate: 1.176E-04 | global batch size:   256 | lm loss: 3.283092E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.116 | TFLOPs: 32.07 |
7:  iteration    83100/  173500 | consumed samples:     21273600 | consumed tokens:  43568332800 | elapsed time per iteration (s): 0.37 | learning rate: 1.174E-04 | global batch size:   256 | lm loss: 3.281754E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.047 | TFLOPs: 32.16 |
7:  iteration    83200/  173500 | consumed samples:     21299200 | consumed tokens:  43620761600 | elapsed time per iteration (s): 0.37 | learning rate: 1.173E-04 | global batch size:   256 | lm loss: 3.279318E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.883 | TFLOPs: 32.01 |
7:  iteration    83300/  173500 | consumed samples:     21324800 | consumed tokens:  43673190400 | elapsed time per iteration (s): 0.37 | learning rate: 1.171E-04 | global batch size:   256 | lm loss: 3.278124E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.022 | TFLOPs: 32.11 |
7:  iteration    83400/  173500 | consumed samples:     21350400 | consumed tokens:  43725619200 | elapsed time per iteration (s): 0.37 | learning rate: 1.169E-04 | global batch size:   256 | lm loss: 3.277982E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.117 | TFLOPs: 32.21 |
7:  iteration    83500/  173500 | consumed samples:     21376000 | consumed tokens:  43778048000 | elapsed time per iteration (s): 0.37 | learning rate: 1.168E-04 | global batch size:   256 | lm loss: 3.281244E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.440 | TFLOPs: 32.23 |
7:  iteration    83600/  173500 | consumed samples:     21401600 | consumed tokens:  43830476800 | elapsed time per iteration (s): 0.37 | learning rate: 1.166E-04 | global batch size:   256 | lm loss: 3.277191E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.186 | TFLOPs: 32.26 |
7:  iteration    83700/  173500 | consumed samples:     21427200 | consumed tokens:  43882905600 | elapsed time per iteration (s): 0.37 | learning rate: 1.164E-04 | global batch size:   256 | lm loss: 3.278846E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.914 | TFLOPs: 32.20 |
7:  iteration    83800/  173500 | consumed samples:     21452800 | consumed tokens:  43935334400 | elapsed time per iteration (s): 0.37 | learning rate: 1.163E-04 | global batch size:   256 | lm loss: 3.281505E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.023 | TFLOPs: 32.25 |
7:  iteration    83900/  173500 | consumed samples:     21478400 | consumed tokens:  43987763200 | elapsed time per iteration (s): 0.37 | learning rate: 1.161E-04 | global batch size:   256 | lm loss: 3.278799E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.318 | TFLOPs: 32.31 |
0: [2023-03-17 08:03:46,404] [INFO] [logging.py:68:log_dist] [Rank 0] step=84000, skipped=0, lr=[0.00011595088621669176, 0.00011595088621669176, 0.00011595088621669176], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    84000/  173500 | consumed samples:     21504000 | consumed tokens:  44040192000 | elapsed time per iteration (s): 0.37 | learning rate: 1.160E-04 | global batch size:   256 | lm loss: 3.276570E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.828 | TFLOPs: 32.11 |
0: steps: 84000 loss: 3.3011 iter time (s): 0.370 samples/sec: 692.729
7:  iteration    84100/  173500 | consumed samples:     21529600 | consumed tokens:  44092620800 | elapsed time per iteration (s): 0.37 | learning rate: 1.158E-04 | global batch size:   256 | lm loss: 3.282056E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.110 | TFLOPs: 32.12 |
7:  iteration    84200/  173500 | consumed samples:     21555200 | consumed tokens:  44145049600 | elapsed time per iteration (s): 0.37 | learning rate: 1.156E-04 | global batch size:   256 | lm loss: 3.275942E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.618 | TFLOPs: 32.10 |
7:  iteration    84300/  173500 | consumed samples:     21580800 | consumed tokens:  44197478400 | elapsed time per iteration (s): 0.37 | learning rate: 1.155E-04 | global batch size:   256 | lm loss: 3.279510E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.791 | TFLOPs: 32.24 |
7:  iteration    84400/  173500 | consumed samples:     21606400 | consumed tokens:  44249907200 | elapsed time per iteration (s): 0.37 | learning rate: 1.153E-04 | global batch size:   256 | lm loss: 3.278881E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.880 | TFLOPs: 32.25 |
7:  iteration    84500/  173500 | consumed samples:     21632000 | consumed tokens:  44302336000 | elapsed time per iteration (s): 0.37 | learning rate: 1.151E-04 | global batch size:   256 | lm loss: 3.283395E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.723 | TFLOPs: 32.33 |
7:  iteration    84600/  173500 | consumed samples:     21657600 | consumed tokens:  44354764800 | elapsed time per iteration (s): 0.37 | learning rate: 1.150E-04 | global batch size:   256 | lm loss: 3.280614E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.530 | TFLOPs: 32.42 |
7:  iteration    84700/  173500 | consumed samples:     21683200 | consumed tokens:  44407193600 | elapsed time per iteration (s): 0.37 | learning rate: 1.148E-04 | global batch size:   256 | lm loss: 3.278877E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.004 | TFLOPs: 32.25 |
7:  iteration    84800/  173500 | consumed samples:     21708800 | consumed tokens:  44459622400 | elapsed time per iteration (s): 0.37 | learning rate: 1.146E-04 | global batch size:   256 | lm loss: 3.277106E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.156 | TFLOPs: 32.40 |
7:  iteration    84900/  173500 | consumed samples:     21734400 | consumed tokens:  44512051200 | elapsed time per iteration (s): 0.37 | learning rate: 1.145E-04 | global batch size:   256 | lm loss: 3.282307E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.125 | TFLOPs: 32.35 |
7:  iteration    85000/  173500 | consumed samples:     21760000 | consumed tokens:  44564480000 | elapsed time per iteration (s): 0.37 | learning rate: 1.143E-04 | global batch size:   256 | lm loss: 3.279356E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.846 | TFLOPs: 32.39 |
7:  iteration    85100/  173500 | consumed samples:     21785600 | consumed tokens:  44616908800 | elapsed time per iteration (s): 0.37 | learning rate: 1.141E-04 | global batch size:   256 | lm loss: 3.276975E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.564 | TFLOPs: 32.42 |
7:  iteration    85200/  173500 | consumed samples:     21811200 | consumed tokens:  44669337600 | elapsed time per iteration (s): 0.37 | learning rate: 1.140E-04 | global batch size:   256 | lm loss: 3.278978E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.192 | TFLOPs: 32.40 |
7:  iteration    85300/  173500 | consumed samples:     21836800 | consumed tokens:  44721766400 | elapsed time per iteration (s): 0.37 | learning rate: 1.138E-04 | global batch size:   256 | lm loss: 3.282954E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.698 | TFLOPs: 32.43 |
7:  iteration    85400/  173500 | consumed samples:     21862400 | consumed tokens:  44774195200 | elapsed time per iteration (s): 0.37 | learning rate: 1.136E-04 | global batch size:   256 | lm loss: 3.279895E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.450 | TFLOPs: 32.41 |
7:  iteration    85500/  173500 | consumed samples:     21888000 | consumed tokens:  44826624000 | elapsed time per iteration (s): 0.37 | learning rate: 1.135E-04 | global batch size:   256 | lm loss: 3.281317E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.685 | TFLOPs: 32.43 |
7:  iteration    85600/  173500 | consumed samples:     21913600 | consumed tokens:  44879052800 | elapsed time per iteration (s): 0.37 | learning rate: 1.133E-04 | global batch size:   256 | lm loss: 3.281777E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.190 | TFLOPs: 32.40 |
7:  iteration    85700/  173500 | consumed samples:     21939200 | consumed tokens:  44931481600 | elapsed time per iteration (s): 0.37 | learning rate: 1.132E-04 | global batch size:   256 | lm loss: 3.274594E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.874 | TFLOPs: 32.43 |
7:  iteration    85800/  173500 | consumed samples:     21964800 | consumed tokens:  44983910400 | elapsed time per iteration (s): 0.37 | learning rate: 1.130E-04 | global batch size:   256 | lm loss: 3.275267E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.129 | TFLOPs: 32.45 |
7:  iteration    85900/  173500 | consumed samples:     21990400 | consumed tokens:  45036339200 | elapsed time per iteration (s): 0.37 | learning rate: 1.128E-04 | global batch size:   256 | lm loss: 3.276920E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.125 | TFLOPs: 32.45 |
0: [2023-03-17 08:16:05,060] [INFO] [logging.py:68:log_dist] [Rank 0] step=86000, skipped=0, lr=[0.0001126626417003261, 0.0001126626417003261, 0.0001126626417003261], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    86000/  173500 | consumed samples:     22016000 | consumed tokens:  45088768000 | elapsed time per iteration (s): 0.37 | learning rate: 1.127E-04 | global batch size:   256 | lm loss: 3.273942E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.485 | TFLOPs: 32.42 |
0: steps: 86000 loss: 3.2995 iter time (s): 0.367 samples/sec: 696.852
7:  iteration    86100/  173500 | consumed samples:     22041600 | consumed tokens:  45141196800 | elapsed time per iteration (s): 0.37 | learning rate: 1.125E-04 | global batch size:   256 | lm loss: 3.278515E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.821 | TFLOPs: 32.43 |
7:  iteration    86200/  173500 | consumed samples:     22067200 | consumed tokens:  45193625600 | elapsed time per iteration (s): 0.37 | learning rate: 1.123E-04 | global batch size:   256 | lm loss: 3.278280E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.184 | TFLOPs: 32.45 |
7:  iteration    86300/  173500 | consumed samples:     22092800 | consumed tokens:  45246054400 | elapsed time per iteration (s): 0.37 | learning rate: 1.122E-04 | global batch size:   256 | lm loss: 3.277183E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.385 | TFLOPs: 32.36 |
7:  iteration    86400/  173500 | consumed samples:     22118400 | consumed tokens:  45298483200 | elapsed time per iteration (s): 0.37 | learning rate: 1.120E-04 | global batch size:   256 | lm loss: 3.269453E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.332 | TFLOPs: 32.41 |
7:  iteration    86500/  173500 | consumed samples:     22144000 | consumed tokens:  45350912000 | elapsed time per iteration (s): 0.37 | learning rate: 1.118E-04 | global batch size:   256 | lm loss: 3.278545E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.374 | TFLOPs: 32.41 |
7:  iteration    86600/  173500 | consumed samples:     22169600 | consumed tokens:  45403340800 | elapsed time per iteration (s): 0.37 | learning rate: 1.117E-04 | global batch size:   256 | lm loss: 3.279554E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.174 | TFLOPs: 32.35 |
7:  iteration    86700/  173500 | consumed samples:     22195200 | consumed tokens:  45455769600 | elapsed time per iteration (s): 0.37 | learning rate: 1.115E-04 | global batch size:   256 | lm loss: 3.280811E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.000 | TFLOPs: 32.44 |
7:  iteration    86800/  173500 | consumed samples:     22220800 | consumed tokens:  45508198400 | elapsed time per iteration (s): 0.37 | learning rate: 1.113E-04 | global batch size:   256 | lm loss: 3.272686E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.778 | TFLOPs: 32.48 |
7:  iteration    86900/  173500 | consumed samples:     22246400 | consumed tokens:  45560627200 | elapsed time per iteration (s): 0.37 | learning rate: 1.112E-04 | global batch size:   256 | lm loss: 3.280152E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.733 | TFLOPs: 32.47 |
7:  iteration    87000/  173500 | consumed samples:     22272000 | consumed tokens:  45613056000 | elapsed time per iteration (s): 0.37 | learning rate: 1.110E-04 | global batch size:   256 | lm loss: 3.270883E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.066 | TFLOPs: 32.44 |
7:  iteration    87100/  173500 | consumed samples:     22297600 | consumed tokens:  45665484800 | elapsed time per iteration (s): 0.37 | learning rate: 1.109E-04 | global batch size:   256 | lm loss: 3.276375E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.888 | TFLOPs: 32.43 |
7:  iteration    87200/  173500 | consumed samples:     22323200 | consumed tokens:  45717913600 | elapsed time per iteration (s): 0.37 | learning rate: 1.107E-04 | global batch size:   256 | lm loss: 3.274551E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.993 | TFLOPs: 32.44 |
7:  iteration    87300/  173500 | consumed samples:     22348800 | consumed tokens:  45770342400 | elapsed time per iteration (s): 0.37 | learning rate: 1.105E-04 | global batch size:   256 | lm loss: 3.273575E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.794 | TFLOPs: 32.43 |
7:  iteration    87400/  173500 | consumed samples:     22374400 | consumed tokens:  45822771200 | elapsed time per iteration (s): 0.37 | learning rate: 1.104E-04 | global batch size:   256 | lm loss: 3.278946E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.544 | TFLOPs: 32.42 |
7:  iteration    87500/  173500 | consumed samples:     22400000 | consumed tokens:  45875200000 | elapsed time per iteration (s): 0.37 | learning rate: 1.102E-04 | global batch size:   256 | lm loss: 3.272906E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.696 | TFLOPs: 32.43 |
7:  iteration    87600/  173500 | consumed samples:     22425600 | consumed tokens:  45927628800 | elapsed time per iteration (s): 0.37 | learning rate: 1.100E-04 | global batch size:   256 | lm loss: 3.275533E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.998 | TFLOPs: 32.44 |
7:  iteration    87700/  173500 | consumed samples:     22451200 | consumed tokens:  45980057600 | elapsed time per iteration (s): 0.37 | learning rate: 1.099E-04 | global batch size:   256 | lm loss: 3.275975E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.693 | TFLOPs: 32.33 |
7:  iteration    87800/  173500 | consumed samples:     22476800 | consumed tokens:  46032486400 | elapsed time per iteration (s): 0.37 | learning rate: 1.097E-04 | global batch size:   256 | lm loss: 3.274172E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.449 | TFLOPs: 32.37 |
7:  iteration    87900/  173500 | consumed samples:     22502400 | consumed tokens:  46084915200 | elapsed time per iteration (s): 0.37 | learning rate: 1.095E-04 | global batch size:   256 | lm loss: 3.272704E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.897 | TFLOPs: 32.39 |
0: [2023-03-17 08:28:22,236] [INFO] [logging.py:68:log_dist] [Rank 0] step=88000, skipped=0, lr=[0.00010937083470846484, 0.00010937083470846484, 0.00010937083470846484], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    88000/  173500 | consumed samples:     22528000 | consumed tokens:  46137344000 | elapsed time per iteration (s): 0.37 | learning rate: 1.094E-04 | global batch size:   256 | lm loss: 3.277990E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.062 | TFLOPs: 32.44 |
0: steps: 88000 loss: 3.3160 iter time (s): 0.367 samples/sec: 698.378
7:  iteration    88100/  173500 | consumed samples:     22553600 | consumed tokens:  46189772800 | elapsed time per iteration (s): 0.37 | learning rate: 1.092E-04 | global batch size:   256 | lm loss: 3.270742E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.573 | TFLOPs: 32.37 |
7:  iteration    88200/  173500 | consumed samples:     22579200 | consumed tokens:  46242201600 | elapsed time per iteration (s): 0.37 | learning rate: 1.090E-04 | global batch size:   256 | lm loss: 3.268668E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.855 | TFLOPs: 32.39 |
7:  iteration    88300/  173500 | consumed samples:     22604800 | consumed tokens:  46294630400 | elapsed time per iteration (s): 0.37 | learning rate: 1.089E-04 | global batch size:   256 | lm loss: 3.274272E+00 | grad norm: 0.239 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.820 | TFLOPs: 32.38 |
7:  iteration    88400/  173500 | consumed samples:     22630400 | consumed tokens:  46347059200 | elapsed time per iteration (s): 0.37 | learning rate: 1.087E-04 | global batch size:   256 | lm loss: 3.274229E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.082 | TFLOPs: 32.40 |
7:  iteration    88500/  173500 | consumed samples:     22656000 | consumed tokens:  46399488000 | elapsed time per iteration (s): 0.37 | learning rate: 1.085E-04 | global batch size:   256 | lm loss: 3.276995E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.001 | TFLOPs: 32.35 |
7:  iteration    88600/  173500 | consumed samples:     22681600 | consumed tokens:  46451916800 | elapsed time per iteration (s): 0.37 | learning rate: 1.084E-04 | global batch size:   256 | lm loss: 3.277179E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.567 | TFLOPs: 32.42 |
7:  iteration    88700/  173500 | consumed samples:     22707200 | consumed tokens:  46504345600 | elapsed time per iteration (s): 0.37 | learning rate: 1.082E-04 | global batch size:   256 | lm loss: 3.274914E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.307 | TFLOPs: 32.36 |
7:  iteration    88800/  173500 | consumed samples:     22732800 | consumed tokens:  46556774400 | elapsed time per iteration (s): 0.37 | learning rate: 1.081E-04 | global batch size:   256 | lm loss: 3.274324E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.375 | TFLOPs: 32.41 |
7:  iteration    88900/  173500 | consumed samples:     22758400 | consumed tokens:  46609203200 | elapsed time per iteration (s): 0.37 | learning rate: 1.079E-04 | global batch size:   256 | lm loss: 3.271926E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.868 | TFLOPs: 32.39 |
7:  iteration    89000/  173500 | consumed samples:     22784000 | consumed tokens:  46661632000 | elapsed time per iteration (s): 0.37 | learning rate: 1.077E-04 | global batch size:   256 | lm loss: 3.269716E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.546 | TFLOPs: 32.37 |
7:  iteration    89100/  173500 | consumed samples:     22809600 | consumed tokens:  46714060800 | elapsed time per iteration (s): 0.37 | learning rate: 1.076E-04 | global batch size:   256 | lm loss: 3.270178E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.941 | TFLOPs: 32.39 |
7:  iteration    89200/  173500 | consumed samples:     22835200 | consumed tokens:  46766489600 | elapsed time per iteration (s): 0.37 | learning rate: 1.074E-04 | global batch size:   256 | lm loss: 3.271279E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.167 | TFLOPs: 32.40 |
7:  iteration    89300/  173500 | consumed samples:     22860800 | consumed tokens:  46818918400 | elapsed time per iteration (s): 0.37 | learning rate: 1.072E-04 | global batch size:   256 | lm loss: 3.265165E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.118 | TFLOPs: 32.40 |
7:  iteration    89400/  173500 | consumed samples:     22886400 | consumed tokens:  46871347200 | elapsed time per iteration (s): 0.37 | learning rate: 1.071E-04 | global batch size:   256 | lm loss: 3.270069E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.613 | TFLOPs: 32.42 |
7:  iteration    89500/  173500 | consumed samples:     22912000 | consumed tokens:  46923776000 | elapsed time per iteration (s): 0.37 | learning rate: 1.069E-04 | global batch size:   256 | lm loss: 3.272387E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.520 | TFLOPs: 32.42 |
7:  iteration    89600/  173500 | consumed samples:     22937600 | consumed tokens:  46976204800 | elapsed time per iteration (s): 0.37 | learning rate: 1.067E-04 | global batch size:   256 | lm loss: 3.271106E+00 | grad norm: 0.197 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.775 | TFLOPs: 32.43 |
7:  iteration    89700/  173500 | consumed samples:     22963200 | consumed tokens:  47028633600 | elapsed time per iteration (s): 0.37 | learning rate: 1.066E-04 | global batch size:   256 | lm loss: 3.270235E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.526 | TFLOPs: 32.46 |
7:  iteration    89800/  173500 | consumed samples:     22988800 | consumed tokens:  47081062400 | elapsed time per iteration (s): 0.37 | learning rate: 1.064E-04 | global batch size:   256 | lm loss: 3.268780E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.590 | TFLOPs: 32.42 |
7:  iteration    89900/  173500 | consumed samples:     23014400 | consumed tokens:  47133491200 | elapsed time per iteration (s): 0.37 | learning rate: 1.062E-04 | global batch size:   256 | lm loss: 3.275505E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.037 | TFLOPs: 32.30 |
0: [2023-03-17 08:40:40,073] [INFO] [logging.py:68:log_dist] [Rank 0] step=90000, skipped=0, lr=[0.00010607986950689534, 0.00010607986950689534, 0.00010607986950689534], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    90000/  173500 | consumed samples:     23040000 | consumed tokens:  47185920000 | elapsed time per iteration (s): 0.37 | learning rate: 1.061E-04 | global batch size:   256 | lm loss: 3.270601E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.131 | TFLOPs: 32.31 |
0: steps: 90000 loss: 3.2487 iter time (s): 0.367 samples/sec: 697.829
7: ------------------------------------------------------------------------------------------------
7: validation loss at iteration 90000 | lm loss value: 3.332263E+00 | lm loss PPL: 2.800163E+01 | 
7: ------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration   90000 to checkpoints_146m91b400m
0: [2023-03-17 08:40:40,200] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step90000 is begin to save!
0: [2023-03-17 08:40:40,204] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_01-model_00-model_states.pt...
0: [2023-03-17 08:40:40,320] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_01-model_00-model_states.pt.
0: [2023-03-17 08:40:40,321] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_03-model_00-model_states.pt...
0: [2023-03-17 08:40:40,336] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_03-model_00-model_states.pt.
0: [2023-03-17 08:40:40,337] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_04-model_00-model_states.pt...
0: [2023-03-17 08:40:40,351] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_04-model_00-model_states.pt.
0: [2023-03-17 08:40:40,352] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_05-model_00-model_states.pt...
0: [2023-03-17 08:40:40,366] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_05-model_00-model_states.pt.
0: [2023-03-17 08:40:40,367] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_06-model_00-model_states.pt...
0: [2023-03-17 08:40:40,381] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_06-model_00-model_states.pt.
0: [2023-03-17 08:40:40,382] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_07-model_00-model_states.pt...
0: [2023-03-17 08:40:40,396] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_07-model_00-model_states.pt.
0: [2023-03-17 08:40:40,397] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_08-model_00-model_states.pt...
0: [2023-03-17 08:40:40,411] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_08-model_00-model_states.pt.
0: [2023-03-17 08:40:40,412] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_09-model_00-model_states.pt...
0: [2023-03-17 08:40:40,426] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_09-model_00-model_states.pt.
0: [2023-03-17 08:40:40,427] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_10-model_00-model_states.pt...
0: [2023-03-17 08:40:40,441] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_10-model_00-model_states.pt.
0: [2023-03-17 08:40:40,442] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_11-model_00-model_states.pt...
0: [2023-03-17 08:40:40,456] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_11-model_00-model_states.pt.
0: [2023-03-17 08:40:40,456] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_12-model_00-model_states.pt...
0: [2023-03-17 08:40:40,471] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_12-model_00-model_states.pt.
0: [2023-03-17 08:40:40,471] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_13-model_00-model_states.pt...
0: [2023-03-17 08:40:40,487] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_13-model_00-model_states.pt.
0: [2023-03-17 08:40:40,487] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_14-model_00-model_states.pt...
0: [2023-03-17 08:40:40,502] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_14-model_00-model_states.pt.
0: [2023-03-17 08:40:40,502] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_15-model_00-model_states.pt...
0: [2023-03-17 08:40:40,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_15-model_00-model_states.pt.
0: [2023-03-17 08:40:40,517] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_16-model_00-model_states.pt...
0: [2023-03-17 08:40:40,532] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_16-model_00-model_states.pt.
0: [2023-03-17 08:40:40,532] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_17-model_00-model_states.pt...
0: [2023-03-17 08:40:40,547] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_17-model_00-model_states.pt.
0: [2023-03-17 08:40:40,547] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/layer_19-model_00-model_states.pt...
0: [2023-03-17 08:40:40,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/layer_19-model_00-model_states.pt.
0: [2023-03-17 08:40:40,549] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step90000/mp_rank_00_model_states.pt
0: [2023-03-17 08:40:40,549] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/mp_rank_00_model_states.pt...
0: [2023-03-17 08:40:40,553] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/mp_rank_00_model_states.pt.
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 08:40:40,570] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
0: [2023-03-17 08:40:40,604] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,606] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,606] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,606] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,608] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,608] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,608] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,610] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,610] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,610] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,610] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,611] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,611] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,611] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,611] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,611] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,612] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,612] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,612] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
6: [2023-03-17 08:40:40,612] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 08:40:40,612] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 08:40:40,612] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,612] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,613] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,613] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,613] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,613] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,613] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
1: [2023-03-17 08:40:40,613] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 08:40:40,614] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 08:40:40,614] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,615] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,615] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,615] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,615] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,620] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,620] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,620] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,623] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,623] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,623] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,623] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
7: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,626] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,626] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,628] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,628] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,629] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,629] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,629] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,629] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
3: [2023-03-17 08:40:40,632] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 08:40:40,632] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 08:40:40,632] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0: [2023-03-17 08:40:40,633] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 08:40:40,633] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
4: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,636] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
5: [2023-03-17 08:40:40,637] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 08:40:40,637] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 08:40:40,637] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step90000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
2: [2023-03-17 08:40:40,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step90000 is ready now!
0:   successfully saved checkpoint at iteration   90000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 460.81
7:  iteration    90100/  173500 | consumed samples:     23065600 | consumed tokens:  47238348800 | elapsed time per iteration (s): 0.37 | learning rate: 1.059E-04 | global batch size:   256 | lm loss: 3.273230E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.866 | TFLOPs: 31.87 |
7:  iteration    90200/  173500 | consumed samples:     23091200 | consumed tokens:  47290777600 | elapsed time per iteration (s): 0.37 | learning rate: 1.058E-04 | global batch size:   256 | lm loss: 3.272161E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.007 | TFLOPs: 32.35 |
7:  iteration    90300/  173500 | consumed samples:     23116800 | consumed tokens:  47343206400 | elapsed time per iteration (s): 0.37 | learning rate: 1.056E-04 | global batch size:   256 | lm loss: 3.269973E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.066 | TFLOPs: 32.35 |
7:  iteration    90400/  173500 | consumed samples:     23142400 | consumed tokens:  47395635200 | elapsed time per iteration (s): 0.37 | learning rate: 1.054E-04 | global batch size:   256 | lm loss: 3.267213E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.164 | TFLOPs: 32.35 |
7:  iteration    90500/  173500 | consumed samples:     23168000 | consumed tokens:  47448064000 | elapsed time per iteration (s): 0.37 | learning rate: 1.053E-04 | global batch size:   256 | lm loss: 3.268781E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.427 | TFLOPs: 32.41 |
7:  iteration    90600/  173500 | consumed samples:     23193600 | consumed tokens:  47500492800 | elapsed time per iteration (s): 0.37 | learning rate: 1.051E-04 | global batch size:   256 | lm loss: 3.275710E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.435 | TFLOPs: 32.37 |
7:  iteration    90700/  173500 | consumed samples:     23219200 | consumed tokens:  47552921600 | elapsed time per iteration (s): 0.37 | learning rate: 1.049E-04 | global batch size:   256 | lm loss: 3.272735E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.949 | TFLOPs: 31.97 |
7:  iteration    90800/  173500 | consumed samples:     23244800 | consumed tokens:  47605350400 | elapsed time per iteration (s): 0.37 | learning rate: 1.048E-04 | global batch size:   256 | lm loss: 3.275305E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.988 | TFLOPs: 32.16 |
7:  iteration    90900/  173500 | consumed samples:     23270400 | consumed tokens:  47657779200 | elapsed time per iteration (s): 0.37 | learning rate: 1.046E-04 | global batch size:   256 | lm loss: 3.272150E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.275 | TFLOPs: 32.03 |
7:  iteration    91000/  173500 | consumed samples:     23296000 | consumed tokens:  47710208000 | elapsed time per iteration (s): 0.37 | learning rate: 1.044E-04 | global batch size:   256 | lm loss: 3.268516E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.861 | TFLOPs: 32.20 |
7:  iteration    91100/  173500 | consumed samples:     23321600 | consumed tokens:  47762636800 | elapsed time per iteration (s): 0.37 | learning rate: 1.043E-04 | global batch size:   256 | lm loss: 3.274294E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.197 | TFLOPs: 32.12 |
7:  iteration    91200/  173500 | consumed samples:     23347200 | consumed tokens:  47815065600 | elapsed time per iteration (s): 0.38 | learning rate: 1.041E-04 | global batch size:   256 | lm loss: 3.266195E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.123 | TFLOPs: 31.75 |
7:  iteration    91300/  173500 | consumed samples:     23372800 | consumed tokens:  47867494400 | elapsed time per iteration (s): 0.38 | learning rate: 1.039E-04 | global batch size:   256 | lm loss: 3.267227E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 677.639 | TFLOPs: 31.63 |
7:  iteration    91400/  173500 | consumed samples:     23398400 | consumed tokens:  47919923200 | elapsed time per iteration (s): 0.38 | learning rate: 1.038E-04 | global batch size:   256 | lm loss: 3.275344E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 677.113 | TFLOPs: 31.61 |
7:  iteration    91500/  173500 | consumed samples:     23424000 | consumed tokens:  47972352000 | elapsed time per iteration (s): 0.38 | learning rate: 1.036E-04 | global batch size:   256 | lm loss: 3.267328E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 677.054 | TFLOPs: 31.60 |
7:  iteration    91600/  173500 | consumed samples:     23449600 | consumed tokens:  48024780800 | elapsed time per iteration (s): 0.37 | learning rate: 1.035E-04 | global batch size:   256 | lm loss: 3.269045E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.097 | TFLOPs: 31.93 |
7:  iteration    91700/  173500 | consumed samples:     23475200 | consumed tokens:  48077209600 | elapsed time per iteration (s): 0.37 | learning rate: 1.033E-04 | global batch size:   256 | lm loss: 3.273712E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.292 | TFLOPs: 31.94 |
7:  iteration    91800/  173500 | consumed samples:     23500800 | consumed tokens:  48129638400 | elapsed time per iteration (s): 0.37 | learning rate: 1.031E-04 | global batch size:   256 | lm loss: 3.271169E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.255 | TFLOPs: 32.03 |
7:  iteration    91900/  173500 | consumed samples:     23526400 | consumed tokens:  48182067200 | elapsed time per iteration (s): 0.37 | learning rate: 1.030E-04 | global batch size:   256 | lm loss: 3.267758E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.828 | TFLOPs: 31.97 |
0: [2023-03-17 08:53:06,175] [INFO] [logging.py:68:log_dist] [Rank 0] step=92000, skipped=0, lr=[0.0001027941492351335, 0.0001027941492351335, 0.0001027941492351335], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    92000/  173500 | consumed samples:     23552000 | consumed tokens:  48234496000 | elapsed time per iteration (s): 0.37 | learning rate: 1.028E-04 | global batch size:   256 | lm loss: 3.271061E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.892 | TFLOPs: 32.01 |
0: steps: 92000 loss: 3.3021 iter time (s): 0.371 samples/sec: 689.486
7:  iteration    92100/  173500 | consumed samples:     23577600 | consumed tokens:  48286924800 | elapsed time per iteration (s): 0.37 | learning rate: 1.026E-04 | global batch size:   256 | lm loss: 3.269195E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.901 | TFLOPs: 32.06 |
7:  iteration    92200/  173500 | consumed samples:     23603200 | consumed tokens:  48339353600 | elapsed time per iteration (s): 0.37 | learning rate: 1.025E-04 | global batch size:   256 | lm loss: 3.270760E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.857 | TFLOPs: 32.20 |
7:  iteration    92300/  173500 | consumed samples:     23628800 | consumed tokens:  48391782400 | elapsed time per iteration (s): 0.37 | learning rate: 1.023E-04 | global batch size:   256 | lm loss: 3.273665E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.638 | TFLOPs: 32.19 |
7:  iteration    92400/  173500 | consumed samples:     23654400 | consumed tokens:  48444211200 | elapsed time per iteration (s): 0.37 | learning rate: 1.021E-04 | global batch size:   256 | lm loss: 3.267630E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.262 | TFLOPs: 32.17 |
7:  iteration    92500/  173500 | consumed samples:     23680000 | consumed tokens:  48496640000 | elapsed time per iteration (s): 0.37 | learning rate: 1.020E-04 | global batch size:   256 | lm loss: 3.270432E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.795 | TFLOPs: 32.01 |
7:  iteration    92600/  173500 | consumed samples:     23705600 | consumed tokens:  48549068800 | elapsed time per iteration (s): 0.37 | learning rate: 1.018E-04 | global batch size:   256 | lm loss: 3.267061E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.659 | TFLOPs: 32.14 |
7:  iteration    92700/  173500 | consumed samples:     23731200 | consumed tokens:  48601497600 | elapsed time per iteration (s): 0.37 | learning rate: 1.016E-04 | global batch size:   256 | lm loss: 3.270463E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.476 | TFLOPs: 32.09 |
7:  iteration    92800/  173500 | consumed samples:     23756800 | consumed tokens:  48653926400 | elapsed time per iteration (s): 0.38 | learning rate: 1.015E-04 | global batch size:   256 | lm loss: 3.271751E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.672 | TFLOPs: 31.44 |
7:  iteration    92900/  173500 | consumed samples:     23782400 | consumed tokens:  48706355200 | elapsed time per iteration (s): 0.37 | learning rate: 1.013E-04 | global batch size:   256 | lm loss: 3.270841E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.270 | TFLOPs: 32.03 |
7:  iteration    93000/  173500 | consumed samples:     23808000 | consumed tokens:  48758784000 | elapsed time per iteration (s): 0.37 | learning rate: 1.012E-04 | global batch size:   256 | lm loss: 3.267463E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.982 | TFLOPs: 32.16 |
7:  iteration    93100/  173500 | consumed samples:     23833600 | consumed tokens:  48811212800 | elapsed time per iteration (s): 0.37 | learning rate: 1.010E-04 | global batch size:   256 | lm loss: 3.268322E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.514 | TFLOPs: 32.28 |
7:  iteration    93200/  173500 | consumed samples:     23859200 | consumed tokens:  48863641600 | elapsed time per iteration (s): 0.37 | learning rate: 1.008E-04 | global batch size:   256 | lm loss: 3.266771E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.856 | TFLOPs: 32.15 |
7:  iteration    93300/  173500 | consumed samples:     23884800 | consumed tokens:  48916070400 | elapsed time per iteration (s): 0.38 | learning rate: 1.007E-04 | global batch size:   256 | lm loss: 3.266013E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.021 | TFLOPs: 31.83 |
7:  iteration    93400/  173500 | consumed samples:     23910400 | consumed tokens:  48968499200 | elapsed time per iteration (s): 0.37 | learning rate: 1.005E-04 | global batch size:   256 | lm loss: 3.267707E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.202 | TFLOPs: 32.03 |
7:  iteration    93500/  173500 | consumed samples:     23936000 | consumed tokens:  49020928000 | elapsed time per iteration (s): 0.37 | learning rate: 1.003E-04 | global batch size:   256 | lm loss: 3.270228E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.530 | TFLOPs: 32.14 |
7:  iteration    93600/  173500 | consumed samples:     23961600 | consumed tokens:  49073356800 | elapsed time per iteration (s): 0.37 | learning rate: 1.002E-04 | global batch size:   256 | lm loss: 3.267497E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.432 | TFLOPs: 32.18 |
7:  iteration    93700/  173500 | consumed samples:     23987200 | consumed tokens:  49125785600 | elapsed time per iteration (s): 0.37 | learning rate: 1.000E-04 | global batch size:   256 | lm loss: 3.261911E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.024 | TFLOPs: 31.97 |
7:  iteration    93800/  173500 | consumed samples:     24012800 | consumed tokens:  49178214400 | elapsed time per iteration (s): 0.37 | learning rate: 9.985E-05 | global batch size:   256 | lm loss: 3.271245E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.027 | TFLOPs: 32.25 |
7:  iteration    93900/  173500 | consumed samples:     24038400 | consumed tokens:  49230643200 | elapsed time per iteration (s): 0.37 | learning rate: 9.968E-05 | global batch size:   256 | lm loss: 3.268051E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.384 | TFLOPs: 32.13 |
0: [2023-03-17 09:05:31,079] [INFO] [logging.py:68:log_dist] [Rank 0] step=94000, skipped=0, lr=[9.951807001525316e-05, 9.951807001525316e-05, 9.951807001525316e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    94000/  173500 | consumed samples:     24064000 | consumed tokens:  49283072000 | elapsed time per iteration (s): 0.37 | learning rate: 9.952E-05 | global batch size:   256 | lm loss: 3.268091E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.666 | TFLOPs: 32.19 |
0: steps: 94000 loss: 3.2532 iter time (s): 0.371 samples/sec: 690.274
7:  iteration    94100/  173500 | consumed samples:     24089600 | consumed tokens:  49335500800 | elapsed time per iteration (s): 0.37 | learning rate: 9.935E-05 | global batch size:   256 | lm loss: 3.267530E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.022 | TFLOPs: 31.93 |
7:  iteration    94200/  173500 | consumed samples:     24115200 | consumed tokens:  49387929600 | elapsed time per iteration (s): 0.37 | learning rate: 9.919E-05 | global batch size:   256 | lm loss: 3.266626E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.416 | TFLOPs: 32.13 |
7:  iteration    94300/  173500 | consumed samples:     24140800 | consumed tokens:  49440358400 | elapsed time per iteration (s): 0.37 | learning rate: 9.903E-05 | global batch size:   256 | lm loss: 3.269504E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.947 | TFLOPs: 32.20 |
7:  iteration    94400/  173500 | consumed samples:     24166400 | consumed tokens:  49492787200 | elapsed time per iteration (s): 0.37 | learning rate: 9.886E-05 | global batch size:   256 | lm loss: 3.263259E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.827 | TFLOPs: 32.25 |
7:  iteration    94500/  173500 | consumed samples:     24192000 | consumed tokens:  49545216000 | elapsed time per iteration (s): 0.37 | learning rate: 9.870E-05 | global batch size:   256 | lm loss: 3.269617E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.285 | TFLOPs: 31.99 |
7:  iteration    94600/  173500 | consumed samples:     24217600 | consumed tokens:  49597644800 | elapsed time per iteration (s): 0.37 | learning rate: 9.854E-05 | global batch size:   256 | lm loss: 3.263629E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.644 | TFLOPs: 32.19 |
7:  iteration    94700/  173500 | consumed samples:     24243200 | consumed tokens:  49650073600 | elapsed time per iteration (s): 0.37 | learning rate: 9.837E-05 | global batch size:   256 | lm loss: 3.265858E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.863 | TFLOPs: 32.29 |
7:  iteration    94800/  173500 | consumed samples:     24268800 | consumed tokens:  49702502400 | elapsed time per iteration (s): 0.37 | learning rate: 9.821E-05 | global batch size:   256 | lm loss: 3.267503E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.841 | TFLOPs: 32.29 |
7:  iteration    94900/  173500 | consumed samples:     24294400 | consumed tokens:  49754931200 | elapsed time per iteration (s): 0.37 | learning rate: 9.805E-05 | global batch size:   256 | lm loss: 3.268262E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.266 | TFLOPs: 32.41 |
7:  iteration    95000/  173500 | consumed samples:     24320000 | consumed tokens:  49807360000 | elapsed time per iteration (s): 0.37 | learning rate: 9.789E-05 | global batch size:   256 | lm loss: 3.263659E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.528 | TFLOPs: 32.46 |
7:  iteration    95100/  173500 | consumed samples:     24345600 | consumed tokens:  49859788800 | elapsed time per iteration (s): 0.37 | learning rate: 9.772E-05 | global batch size:   256 | lm loss: 3.266191E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.563 | TFLOPs: 32.42 |
7:  iteration    95200/  173500 | consumed samples:     24371200 | consumed tokens:  49912217600 | elapsed time per iteration (s): 0.37 | learning rate: 9.756E-05 | global batch size:   256 | lm loss: 3.263964E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.141 | TFLOPs: 32.40 |
7:  iteration    95300/  173500 | consumed samples:     24396800 | consumed tokens:  49964646400 | elapsed time per iteration (s): 0.37 | learning rate: 9.740E-05 | global batch size:   256 | lm loss: 3.265111E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.840 | TFLOPs: 32.43 |
7:  iteration    95400/  173500 | consumed samples:     24422400 | consumed tokens:  50017075200 | elapsed time per iteration (s): 0.37 | learning rate: 9.723E-05 | global batch size:   256 | lm loss: 3.266839E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.064 | TFLOPs: 32.35 |
7:  iteration    95500/  173500 | consumed samples:     24448000 | consumed tokens:  50069504000 | elapsed time per iteration (s): 0.37 | learning rate: 9.707E-05 | global batch size:   256 | lm loss: 3.265894E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.153 | TFLOPs: 32.40 |
7:  iteration    95600/  173500 | consumed samples:     24473600 | consumed tokens:  50121932800 | elapsed time per iteration (s): 0.37 | learning rate: 9.691E-05 | global batch size:   256 | lm loss: 3.264485E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.466 | TFLOPs: 32.42 |
7:  iteration    95700/  173500 | consumed samples:     24499200 | consumed tokens:  50174361600 | elapsed time per iteration (s): 0.37 | learning rate: 9.674E-05 | global batch size:   256 | lm loss: 3.266021E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.275 | TFLOPs: 32.36 |
7:  iteration    95800/  173500 | consumed samples:     24524800 | consumed tokens:  50226790400 | elapsed time per iteration (s): 0.37 | learning rate: 9.658E-05 | global batch size:   256 | lm loss: 3.268237E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.543 | TFLOPs: 32.42 |
7:  iteration    95900/  173500 | consumed samples:     24550400 | consumed tokens:  50279219200 | elapsed time per iteration (s): 0.37 | learning rate: 9.642E-05 | global batch size:   256 | lm loss: 3.270068E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.215 | TFLOPs: 32.45 |
0: [2023-03-17 09:17:50,714] [INFO] [logging.py:68:log_dist] [Rank 0] step=96000, skipped=0, lr=[9.625601507010446e-05, 9.625601507010446e-05, 9.625601507010446e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    96000/  173500 | consumed samples:     24576000 | consumed tokens:  50331648000 | elapsed time per iteration (s): 0.37 | learning rate: 9.626E-05 | global batch size:   256 | lm loss: 3.265933E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.095 | TFLOPs: 32.44 |
0: steps: 96000 loss: 3.2621 iter time (s): 0.368 samples/sec: 696.149
7:  iteration    96100/  173500 | consumed samples:     24601600 | consumed tokens:  50384076800 | elapsed time per iteration (s): 0.37 | learning rate: 9.609E-05 | global batch size:   256 | lm loss: 3.265157E+00 | grad norm: 0.203 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.166 | TFLOPs: 32.40 |
7:  iteration    96200/  173500 | consumed samples:     24627200 | consumed tokens:  50436505600 | elapsed time per iteration (s): 0.37 | learning rate: 9.593E-05 | global batch size:   256 | lm loss: 3.262776E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.468 | TFLOPs: 32.37 |
7:  iteration    96300/  173500 | consumed samples:     24652800 | consumed tokens:  50488934400 | elapsed time per iteration (s): 0.37 | learning rate: 9.577E-05 | global batch size:   256 | lm loss: 3.262650E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.848 | TFLOPs: 32.43 |
7:  iteration    96400/  173500 | consumed samples:     24678400 | consumed tokens:  50541363200 | elapsed time per iteration (s): 0.37 | learning rate: 9.561E-05 | global batch size:   256 | lm loss: 3.262460E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.038 | TFLOPs: 32.40 |
7:  iteration    96500/  173500 | consumed samples:     24704000 | consumed tokens:  50593792000 | elapsed time per iteration (s): 0.37 | learning rate: 9.544E-05 | global batch size:   256 | lm loss: 3.264705E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.647 | TFLOPs: 32.42 |
7:  iteration    96600/  173500 | consumed samples:     24729600 | consumed tokens:  50646220800 | elapsed time per iteration (s): 0.37 | learning rate: 9.528E-05 | global batch size:   256 | lm loss: 3.265872E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.330 | TFLOPs: 32.41 |
7:  iteration    96700/  173500 | consumed samples:     24755200 | consumed tokens:  50698649600 | elapsed time per iteration (s): 0.37 | learning rate: 9.512E-05 | global batch size:   256 | lm loss: 3.269145E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.726 | TFLOPs: 32.43 |
7:  iteration    96800/  173500 | consumed samples:     24780800 | consumed tokens:  50751078400 | elapsed time per iteration (s): 0.37 | learning rate: 9.496E-05 | global batch size:   256 | lm loss: 3.264930E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.663 | TFLOPs: 32.42 |
7:  iteration    96900/  173500 | consumed samples:     24806400 | consumed tokens:  50803507200 | elapsed time per iteration (s): 0.37 | learning rate: 9.479E-05 | global batch size:   256 | lm loss: 3.263508E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.665 | TFLOPs: 32.42 |
7:  iteration    97000/  173500 | consumed samples:     24832000 | consumed tokens:  50855936000 | elapsed time per iteration (s): 0.37 | learning rate: 9.463E-05 | global batch size:   256 | lm loss: 3.268937E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.881 | TFLOPs: 32.43 |
7:  iteration    97100/  173500 | consumed samples:     24857600 | consumed tokens:  50908364800 | elapsed time per iteration (s): 0.37 | learning rate: 9.447E-05 | global batch size:   256 | lm loss: 3.270002E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.841 | TFLOPs: 32.43 |
7:  iteration    97200/  173500 | consumed samples:     24883200 | consumed tokens:  50960793600 | elapsed time per iteration (s): 0.37 | learning rate: 9.431E-05 | global batch size:   256 | lm loss: 3.256913E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.416 | TFLOPs: 32.37 |
7:  iteration    97300/  173500 | consumed samples:     24908800 | consumed tokens:  51013222400 | elapsed time per iteration (s): 0.37 | learning rate: 9.415E-05 | global batch size:   256 | lm loss: 3.268459E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.339 | TFLOPs: 32.41 |
7:  iteration    97400/  173500 | consumed samples:     24934400 | consumed tokens:  51065651200 | elapsed time per iteration (s): 0.37 | learning rate: 9.398E-05 | global batch size:   256 | lm loss: 3.262606E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.898 | TFLOPs: 32.44 |
7:  iteration    97500/  173500 | consumed samples:     24960000 | consumed tokens:  51118080000 | elapsed time per iteration (s): 0.37 | learning rate: 9.382E-05 | global batch size:   256 | lm loss: 3.263836E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.995 | TFLOPs: 32.44 |
7:  iteration    97600/  173500 | consumed samples:     24985600 | consumed tokens:  51170508800 | elapsed time per iteration (s): 0.37 | learning rate: 9.366E-05 | global batch size:   256 | lm loss: 3.259240E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.589 | TFLOPs: 32.42 |
7:  iteration    97700/  173500 | consumed samples:     25011200 | consumed tokens:  51222937600 | elapsed time per iteration (s): 0.37 | learning rate: 9.350E-05 | global batch size:   256 | lm loss: 3.262779E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.808 | TFLOPs: 32.43 |
7:  iteration    97800/  173500 | consumed samples:     25036800 | consumed tokens:  51275366400 | elapsed time per iteration (s): 0.37 | learning rate: 9.334E-05 | global batch size:   256 | lm loss: 3.262115E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.005 | TFLOPs: 32.39 |
7:  iteration    97900/  173500 | consumed samples:     25062400 | consumed tokens:  51327795200 | elapsed time per iteration (s): 0.37 | learning rate: 9.317E-05 | global batch size:   256 | lm loss: 3.264597E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.472 | TFLOPs: 32.42 |
0: [2023-03-17 09:30:07,945] [INFO] [logging.py:68:log_dist] [Rank 0] step=98000, skipped=0, lr=[9.301234885879047e-05, 9.301234885879047e-05, 9.301234885879047e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration    98000/  173500 | consumed samples:     25088000 | consumed tokens:  51380224000 | elapsed time per iteration (s): 0.37 | learning rate: 9.301E-05 | global batch size:   256 | lm loss: 3.261623E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.019 | TFLOPs: 32.44 |
0: steps: 98000 loss: 3.2602 iter time (s): 0.367 samples/sec: 698.241
7:  iteration    98100/  173500 | consumed samples:     25113600 | consumed tokens:  51432652800 | elapsed time per iteration (s): 0.37 | learning rate: 9.285E-05 | global batch size:   256 | lm loss: 3.264355E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.701 | TFLOPs: 32.43 |
7:  iteration    98200/  173500 | consumed samples:     25139200 | consumed tokens:  51485081600 | elapsed time per iteration (s): 0.37 | learning rate: 9.269E-05 | global batch size:   256 | lm loss: 3.262003E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.669 | TFLOPs: 32.42 |
7:  iteration    98300/  173500 | consumed samples:     25164800 | consumed tokens:  51537510400 | elapsed time per iteration (s): 0.37 | learning rate: 9.253E-05 | global batch size:   256 | lm loss: 3.257898E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.783 | TFLOPs: 32.43 |
7:  iteration    98400/  173500 | consumed samples:     25190400 | consumed tokens:  51589939200 | elapsed time per iteration (s): 0.37 | learning rate: 9.237E-05 | global batch size:   256 | lm loss: 3.261681E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.444 | TFLOPs: 32.37 |
7:  iteration    98500/  173500 | consumed samples:     25216000 | consumed tokens:  51642368000 | elapsed time per iteration (s): 0.37 | learning rate: 9.220E-05 | global batch size:   256 | lm loss: 3.256642E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.261 | TFLOPs: 32.22 |
7:  iteration    98600/  173500 | consumed samples:     25241600 | consumed tokens:  51694796800 | elapsed time per iteration (s): 0.37 | learning rate: 9.204E-05 | global batch size:   256 | lm loss: 3.265215E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.219 | TFLOPs: 32.31 |
7:  iteration    98700/  173500 | consumed samples:     25267200 | consumed tokens:  51747225600 | elapsed time per iteration (s): 0.37 | learning rate: 9.188E-05 | global batch size:   256 | lm loss: 3.262101E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.850 | TFLOPs: 32.39 |
7:  iteration    98800/  173500 | consumed samples:     25292800 | consumed tokens:  51799654400 | elapsed time per iteration (s): 0.37 | learning rate: 9.172E-05 | global batch size:   256 | lm loss: 3.261451E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.065 | TFLOPs: 32.44 |
7:  iteration    98900/  173500 | consumed samples:     25318400 | consumed tokens:  51852083200 | elapsed time per iteration (s): 0.37 | learning rate: 9.156E-05 | global batch size:   256 | lm loss: 3.262448E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.020 | TFLOPs: 32.44 |
7:  iteration    99000/  173500 | consumed samples:     25344000 | consumed tokens:  51904512000 | elapsed time per iteration (s): 0.37 | learning rate: 9.140E-05 | global batch size:   256 | lm loss: 3.257117E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.830 | TFLOPs: 32.43 |
7:  iteration    99100/  173500 | consumed samples:     25369600 | consumed tokens:  51956940800 | elapsed time per iteration (s): 0.37 | learning rate: 9.124E-05 | global batch size:   256 | lm loss: 3.265577E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.990 | TFLOPs: 32.44 |
7:  iteration    99200/  173500 | consumed samples:     25395200 | consumed tokens:  52009369600 | elapsed time per iteration (s): 0.37 | learning rate: 9.108E-05 | global batch size:   256 | lm loss: 3.262440E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.882 | TFLOPs: 32.43 |
7:  iteration    99300/  173500 | consumed samples:     25420800 | consumed tokens:  52061798400 | elapsed time per iteration (s): 0.37 | learning rate: 9.092E-05 | global batch size:   256 | lm loss: 3.262299E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.788 | TFLOPs: 32.43 |
7:  iteration    99400/  173500 | consumed samples:     25446400 | consumed tokens:  52114227200 | elapsed time per iteration (s): 0.37 | learning rate: 9.076E-05 | global batch size:   256 | lm loss: 3.264816E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.284 | TFLOPs: 32.45 |
7:  iteration    99500/  173500 | consumed samples:     25472000 | consumed tokens:  52166656000 | elapsed time per iteration (s): 0.37 | learning rate: 9.059E-05 | global batch size:   256 | lm loss: 3.263208E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.701 | TFLOPs: 32.43 |
7:  iteration    99600/  173500 | consumed samples:     25497600 | consumed tokens:  52219084800 | elapsed time per iteration (s): 0.37 | learning rate: 9.043E-05 | global batch size:   256 | lm loss: 3.258603E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.541 | TFLOPs: 32.42 |
7:  iteration    99700/  173500 | consumed samples:     25523200 | consumed tokens:  52271513600 | elapsed time per iteration (s): 0.37 | learning rate: 9.027E-05 | global batch size:   256 | lm loss: 3.265228E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.822 | TFLOPs: 32.43 |
7:  iteration    99800/  173500 | consumed samples:     25548800 | consumed tokens:  52323942400 | elapsed time per iteration (s): 0.37 | learning rate: 9.011E-05 | global batch size:   256 | lm loss: 3.262363E+00 | grad norm: 0.202 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.012 | TFLOPs: 32.44 |
7:  iteration    99900/  173500 | consumed samples:     25574400 | consumed tokens:  52376371200 | elapsed time per iteration (s): 0.37 | learning rate: 8.995E-05 | global batch size:   256 | lm loss: 3.266832E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.107 | TFLOPs: 32.45 |
0: [2023-03-17 09:42:25,304] [INFO] [logging.py:68:log_dist] [Rank 0] step=100000, skipped=0, lr=[8.979141123724914e-05, 8.979141123724914e-05, 8.979141123724914e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   100000/  173500 | consumed samples:     25600000 | consumed tokens:  52428800000 | elapsed time per iteration (s): 0.37 | learning rate: 8.979E-05 | global batch size:   256 | lm loss: 3.260991E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.489 | TFLOPs: 32.42 |
0: steps: 100000 loss: 3.2662 iter time (s): 0.367 samples/sec: 698.153
7: -------------------------------------------------------------------------------------------------
7: validation loss at iteration 100000 | lm loss value: 3.283149E+00 | lm loss PPL: 2.665958E+01 | 
7: -------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration  100000 to checkpoints_146m91b400m
0: [2023-03-17 09:42:25,432] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step100000 is begin to save!
0: [2023-03-17 09:42:25,437] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_01-model_00-model_states.pt...
0: [2023-03-17 09:42:25,556] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_01-model_00-model_states.pt.
0: [2023-03-17 09:42:25,557] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_03-model_00-model_states.pt...
0: [2023-03-17 09:42:25,572] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_03-model_00-model_states.pt.
0: [2023-03-17 09:42:25,572] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_04-model_00-model_states.pt...
0: [2023-03-17 09:42:25,587] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_04-model_00-model_states.pt.
0: [2023-03-17 09:42:25,587] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_05-model_00-model_states.pt...
0: [2023-03-17 09:42:25,602] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_05-model_00-model_states.pt.
0: [2023-03-17 09:42:25,602] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_06-model_00-model_states.pt...
0: [2023-03-17 09:42:25,617] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_06-model_00-model_states.pt.
0: [2023-03-17 09:42:25,617] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_07-model_00-model_states.pt...
0: [2023-03-17 09:42:25,632] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_07-model_00-model_states.pt.
0: [2023-03-17 09:42:25,632] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_08-model_00-model_states.pt...
0: [2023-03-17 09:42:25,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_08-model_00-model_states.pt.
0: [2023-03-17 09:42:25,647] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_09-model_00-model_states.pt...
0: [2023-03-17 09:42:25,662] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_09-model_00-model_states.pt.
0: [2023-03-17 09:42:25,662] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_10-model_00-model_states.pt...
0: [2023-03-17 09:42:25,677] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_10-model_00-model_states.pt.
0: [2023-03-17 09:42:25,677] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_11-model_00-model_states.pt...
0: [2023-03-17 09:42:25,692] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_11-model_00-model_states.pt.
0: [2023-03-17 09:42:25,692] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_12-model_00-model_states.pt...
0: [2023-03-17 09:42:25,707] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_12-model_00-model_states.pt.
0: [2023-03-17 09:42:25,707] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_13-model_00-model_states.pt...
0: [2023-03-17 09:42:25,722] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_13-model_00-model_states.pt.
0: [2023-03-17 09:42:25,722] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_14-model_00-model_states.pt...
0: [2023-03-17 09:42:25,736] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_14-model_00-model_states.pt.
0: [2023-03-17 09:42:25,737] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_15-model_00-model_states.pt...
0: [2023-03-17 09:42:25,751] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_15-model_00-model_states.pt.
0: [2023-03-17 09:42:25,752] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_16-model_00-model_states.pt...
0: [2023-03-17 09:42:25,766] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_16-model_00-model_states.pt.
0: [2023-03-17 09:42:25,767] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_17-model_00-model_states.pt...
0: [2023-03-17 09:42:25,781] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_17-model_00-model_states.pt.
0: [2023-03-17 09:42:25,782] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/layer_19-model_00-model_states.pt...
0: [2023-03-17 09:42:25,783] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/layer_19-model_00-model_states.pt.
0: [2023-03-17 09:42:25,783] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step100000/mp_rank_00_model_states.pt
0: [2023-03-17 09:42:25,783] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/mp_rank_00_model_states.pt...
0: [2023-03-17 09:42:25,787] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/mp_rank_00_model_states.pt.
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 09:42:25,805] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
0: [2023-03-17 09:42:25,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,841] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,841] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,841] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,842] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,842] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,842] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,844] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,844] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,844] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,844] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,844] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,844] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,844] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
0: [2023-03-17 09:42:25,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,846] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,846] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,846] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,855] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,855] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 09:42:25,856] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,856] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,856] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
7: [2023-03-17 09:42:25,856] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,857] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,858] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,858] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 09:42:25,858] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,858] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 09:42:25,858] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
1: [2023-03-17 09:42:25,858] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
6: [2023-03-17 09:42:25,859] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 09:42:25,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 09:42:25,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0: [2023-03-17 09:42:25,873] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,873] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 09:42:25,874] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
3: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
5: [2023-03-17 09:42:25,874] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
3: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 09:42:25,876] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,876] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
2: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,877] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,877] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,878] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,878] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,878] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,878] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,878] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
4: [2023-03-17 09:42:25,879] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 09:42:25,879] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step100000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 09:42:25,879] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step100000 is ready now!
0:   successfully saved checkpoint at iteration  100000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 458.88
7:  iteration   100100/  173500 | consumed samples:     25625600 | consumed tokens:  52481228800 | elapsed time per iteration (s): 0.37 | learning rate: 8.963E-05 | global batch size:   256 | lm loss: 3.260518E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.550 | TFLOPs: 31.95 |
7:  iteration   100200/  173500 | consumed samples:     25651200 | consumed tokens:  52533657600 | elapsed time per iteration (s): 0.37 | learning rate: 8.947E-05 | global batch size:   256 | lm loss: 3.258515E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.559 | TFLOPs: 32.37 |
7:  iteration   100300/  173500 | consumed samples:     25676800 | consumed tokens:  52586086400 | elapsed time per iteration (s): 0.37 | learning rate: 8.931E-05 | global batch size:   256 | lm loss: 3.257861E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.550 | TFLOPs: 32.42 |
7:  iteration   100400/  173500 | consumed samples:     25702400 | consumed tokens:  52638515200 | elapsed time per iteration (s): 0.37 | learning rate: 8.915E-05 | global batch size:   256 | lm loss: 3.260352E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.074 | TFLOPs: 32.44 |
7:  iteration   100500/  173500 | consumed samples:     25728000 | consumed tokens:  52690944000 | elapsed time per iteration (s): 0.37 | learning rate: 8.899E-05 | global batch size:   256 | lm loss: 3.260696E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.836 | TFLOPs: 32.43 |
7:  iteration   100600/  173500 | consumed samples:     25753600 | consumed tokens:  52743372800 | elapsed time per iteration (s): 0.37 | learning rate: 8.883E-05 | global batch size:   256 | lm loss: 3.261532E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.398 | TFLOPs: 32.41 |
7:  iteration   100700/  173500 | consumed samples:     25779200 | consumed tokens:  52795801600 | elapsed time per iteration (s): 0.37 | learning rate: 8.867E-05 | global batch size:   256 | lm loss: 3.256365E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.446 | TFLOPs: 32.23 |
7:  iteration   100800/  173500 | consumed samples:     25804800 | consumed tokens:  52848230400 | elapsed time per iteration (s): 0.37 | learning rate: 8.851E-05 | global batch size:   256 | lm loss: 3.258783E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.400 | TFLOPs: 32.46 |
7:  iteration   100900/  173500 | consumed samples:     25830400 | consumed tokens:  52900659200 | elapsed time per iteration (s): 0.37 | learning rate: 8.835E-05 | global batch size:   256 | lm loss: 3.261281E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.777 | TFLOPs: 32.43 |
7:  iteration   101000/  173500 | consumed samples:     25856000 | consumed tokens:  52953088000 | elapsed time per iteration (s): 0.37 | learning rate: 8.819E-05 | global batch size:   256 | lm loss: 3.260405E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.246 | TFLOPs: 32.45 |
7:  iteration   101100/  173500 | consumed samples:     25881600 | consumed tokens:  53005516800 | elapsed time per iteration (s): 0.37 | learning rate: 8.803E-05 | global batch size:   256 | lm loss: 3.258645E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.665 | TFLOPs: 32.38 |
7:  iteration   101200/  173500 | consumed samples:     25907200 | consumed tokens:  53057945600 | elapsed time per iteration (s): 0.37 | learning rate: 8.787E-05 | global batch size:   256 | lm loss: 3.260112E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.782 | TFLOPs: 32.34 |
7:  iteration   101300/  173500 | consumed samples:     25932800 | consumed tokens:  53110374400 | elapsed time per iteration (s): 0.37 | learning rate: 8.771E-05 | global batch size:   256 | lm loss: 3.260526E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.163 | TFLOPs: 32.12 |
7:  iteration   101400/  173500 | consumed samples:     25958400 | consumed tokens:  53162803200 | elapsed time per iteration (s): 0.37 | learning rate: 8.755E-05 | global batch size:   256 | lm loss: 3.258130E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.010 | TFLOPs: 32.11 |
7:  iteration   101500/  173500 | consumed samples:     25984000 | consumed tokens:  53215232000 | elapsed time per iteration (s): 0.37 | learning rate: 8.739E-05 | global batch size:   256 | lm loss: 3.257044E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.645 | TFLOPs: 32.33 |
7:  iteration   101600/  173500 | consumed samples:     26009600 | consumed tokens:  53267660800 | elapsed time per iteration (s): 0.37 | learning rate: 8.723E-05 | global batch size:   256 | lm loss: 3.262505E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.115 | TFLOPs: 32.21 |
7:  iteration   101700/  173500 | consumed samples:     26035200 | consumed tokens:  53320089600 | elapsed time per iteration (s): 0.37 | learning rate: 8.707E-05 | global batch size:   256 | lm loss: 3.258432E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.019 | TFLOPs: 31.97 |
7:  iteration   101800/  173500 | consumed samples:     26060800 | consumed tokens:  53372518400 | elapsed time per iteration (s): 0.37 | learning rate: 8.692E-05 | global batch size:   256 | lm loss: 3.259686E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.930 | TFLOPs: 32.16 |
7:  iteration   101900/  173500 | consumed samples:     26086400 | consumed tokens:  53424947200 | elapsed time per iteration (s): 0.37 | learning rate: 8.676E-05 | global batch size:   256 | lm loss: 3.259324E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.526 | TFLOPs: 32.09 |
0: [2023-03-17 09:54:46,178] [INFO] [logging.py:68:log_dist] [Rank 0] step=102000, skipped=0, lr=[8.659751165175261e-05, 8.659751165175261e-05, 8.659751165175261e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   102000/  173500 | consumed samples:     26112000 | consumed tokens:  53477376000 | elapsed time per iteration (s): 0.38 | learning rate: 8.660E-05 | global batch size:   256 | lm loss: 3.263854E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.289 | TFLOPs: 31.85 |
0: steps: 102000 loss: 3.2798 iter time (s): 0.368 samples/sec: 695.292
7:  iteration   102100/  173500 | consumed samples:     26137600 | consumed tokens:  53529804800 | elapsed time per iteration (s): 0.37 | learning rate: 8.644E-05 | global batch size:   256 | lm loss: 3.262780E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.823 | TFLOPs: 32.20 |
7:  iteration   102200/  173500 | consumed samples:     26163200 | consumed tokens:  53582233600 | elapsed time per iteration (s): 0.37 | learning rate: 8.628E-05 | global batch size:   256 | lm loss: 3.255223E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.349 | TFLOPs: 32.18 |
7:  iteration   102300/  173500 | consumed samples:     26188800 | consumed tokens:  53634662400 | elapsed time per iteration (s): 0.37 | learning rate: 8.612E-05 | global batch size:   256 | lm loss: 3.256172E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.370 | TFLOPs: 32.41 |
7:  iteration   102400/  173500 | consumed samples:     26214400 | consumed tokens:  53687091200 | elapsed time per iteration (s): 0.37 | learning rate: 8.596E-05 | global batch size:   256 | lm loss: 3.261361E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.491 | TFLOPs: 32.42 |
7:  iteration   102500/  173500 | consumed samples:     26240000 | consumed tokens:  53739520000 | elapsed time per iteration (s): 0.37 | learning rate: 8.580E-05 | global batch size:   256 | lm loss: 3.261457E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.720 | TFLOPs: 32.43 |
7:  iteration   102600/  173500 | consumed samples:     26265600 | consumed tokens:  53791948800 | elapsed time per iteration (s): 0.37 | learning rate: 8.565E-05 | global batch size:   256 | lm loss: 3.258513E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.965 | TFLOPs: 32.44 |
7:  iteration   102700/  173500 | consumed samples:     26291200 | consumed tokens:  53844377600 | elapsed time per iteration (s): 0.37 | learning rate: 8.549E-05 | global batch size:   256 | lm loss: 3.253723E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.891 | TFLOPs: 32.43 |
7:  iteration   102800/  173500 | consumed samples:     26316800 | consumed tokens:  53896806400 | elapsed time per iteration (s): 0.37 | learning rate: 8.533E-05 | global batch size:   256 | lm loss: 3.257420E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.623 | TFLOPs: 32.38 |
7:  iteration   102900/  173500 | consumed samples:     26342400 | consumed tokens:  53949235200 | elapsed time per iteration (s): 0.37 | learning rate: 8.517E-05 | global batch size:   256 | lm loss: 3.261965E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.409 | TFLOPs: 32.46 |
7:  iteration   103000/  173500 | consumed samples:     26368000 | consumed tokens:  54001664000 | elapsed time per iteration (s): 0.37 | learning rate: 8.501E-05 | global batch size:   256 | lm loss: 3.254794E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.268 | TFLOPs: 32.45 |
7:  iteration   103100/  173500 | consumed samples:     26393600 | consumed tokens:  54054092800 | elapsed time per iteration (s): 0.37 | learning rate: 8.485E-05 | global batch size:   256 | lm loss: 3.255371E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.985 | TFLOPs: 32.44 |
7:  iteration   103200/  173500 | consumed samples:     26419200 | consumed tokens:  54106521600 | elapsed time per iteration (s): 0.37 | learning rate: 8.470E-05 | global batch size:   256 | lm loss: 3.261209E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.940 | TFLOPs: 32.44 |
7:  iteration   103300/  173500 | consumed samples:     26444800 | consumed tokens:  54158950400 | elapsed time per iteration (s): 0.37 | learning rate: 8.454E-05 | global batch size:   256 | lm loss: 3.257004E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.105 | TFLOPs: 32.44 |
7:  iteration   103400/  173500 | consumed samples:     26470400 | consumed tokens:  54211379200 | elapsed time per iteration (s): 0.37 | learning rate: 8.438E-05 | global batch size:   256 | lm loss: 3.257890E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.512 | TFLOPs: 32.23 |
7:  iteration   103500/  173500 | consumed samples:     26496000 | consumed tokens:  54263808000 | elapsed time per iteration (s): 0.37 | learning rate: 8.422E-05 | global batch size:   256 | lm loss: 3.258625E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.559 | TFLOPs: 32.23 |
7:  iteration   103600/  173500 | consumed samples:     26521600 | consumed tokens:  54316236800 | elapsed time per iteration (s): 0.37 | learning rate: 8.406E-05 | global batch size:   256 | lm loss: 3.259803E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.553 | TFLOPs: 32.47 |
7:  iteration   103700/  173500 | consumed samples:     26547200 | consumed tokens:  54368665600 | elapsed time per iteration (s): 0.37 | learning rate: 8.391E-05 | global batch size:   256 | lm loss: 3.258707E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.091 | TFLOPs: 32.44 |
7:  iteration   103800/  173500 | consumed samples:     26572800 | consumed tokens:  54421094400 | elapsed time per iteration (s): 0.37 | learning rate: 8.375E-05 | global batch size:   256 | lm loss: 3.259405E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.614 | TFLOPs: 32.47 |
7:  iteration   103900/  173500 | consumed samples:     26598400 | consumed tokens:  54473523200 | elapsed time per iteration (s): 0.37 | learning rate: 8.359E-05 | global batch size:   256 | lm loss: 3.259555E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.372 | TFLOPs: 32.46 |
0: [2023-03-17 10:07:03,944] [INFO] [logging.py:68:log_dist] [Rank 0] step=104000, skipped=0, lr=[8.343492337309329e-05, 8.343492337309329e-05, 8.343492337309329e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   104000/  173500 | consumed samples:     26624000 | consumed tokens:  54525952000 | elapsed time per iteration (s): 0.37 | learning rate: 8.343E-05 | global batch size:   256 | lm loss: 3.258615E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.194 | TFLOPs: 32.45 |
0: steps: 104000 loss: 3.2476 iter time (s): 0.367 samples/sec: 697.938
7:  iteration   104100/  173500 | consumed samples:     26649600 | consumed tokens:  54578380800 | elapsed time per iteration (s): 0.37 | learning rate: 8.328E-05 | global batch size:   256 | lm loss: 3.264011E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.020 | TFLOPs: 32.44 |
7:  iteration   104200/  173500 | consumed samples:     26675200 | consumed tokens:  54630809600 | elapsed time per iteration (s): 0.37 | learning rate: 8.312E-05 | global batch size:   256 | lm loss: 3.258285E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.505 | TFLOPs: 32.46 |
7:  iteration   104300/  173500 | consumed samples:     26700800 | consumed tokens:  54683238400 | elapsed time per iteration (s): 0.37 | learning rate: 8.296E-05 | global batch size:   256 | lm loss: 3.258074E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.690 | TFLOPs: 32.47 |
7:  iteration   104400/  173500 | consumed samples:     26726400 | consumed tokens:  54735667200 | elapsed time per iteration (s): 0.37 | learning rate: 8.281E-05 | global batch size:   256 | lm loss: 3.257722E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.675 | TFLOPs: 32.47 |
7:  iteration   104500/  173500 | consumed samples:     26752000 | consumed tokens:  54788096000 | elapsed time per iteration (s): 0.37 | learning rate: 8.265E-05 | global batch size:   256 | lm loss: 3.260758E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.593 | TFLOPs: 32.47 |
7:  iteration   104600/  173500 | consumed samples:     26777600 | consumed tokens:  54840524800 | elapsed time per iteration (s): 0.37 | learning rate: 8.249E-05 | global batch size:   256 | lm loss: 3.257681E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.411 | TFLOPs: 32.41 |
7:  iteration   104700/  173500 | consumed samples:     26803200 | consumed tokens:  54892953600 | elapsed time per iteration (s): 0.37 | learning rate: 8.234E-05 | global batch size:   256 | lm loss: 3.255555E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 696.005 | TFLOPs: 32.49 |
7:  iteration   104800/  173500 | consumed samples:     26828800 | consumed tokens:  54945382400 | elapsed time per iteration (s): 0.37 | learning rate: 8.218E-05 | global batch size:   256 | lm loss: 3.257251E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.886 | TFLOPs: 32.43 |
7:  iteration   104900/  173500 | consumed samples:     26854400 | consumed tokens:  54997811200 | elapsed time per iteration (s): 0.37 | learning rate: 8.202E-05 | global batch size:   256 | lm loss: 3.255072E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.096 | TFLOPs: 32.44 |
7:  iteration   105000/  173500 | consumed samples:     26880000 | consumed tokens:  55050240000 | elapsed time per iteration (s): 0.37 | learning rate: 8.187E-05 | global batch size:   256 | lm loss: 3.255745E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.184 | TFLOPs: 32.22 |
7:  iteration   105100/  173500 | consumed samples:     26905600 | consumed tokens:  55102668800 | elapsed time per iteration (s): 0.37 | learning rate: 8.171E-05 | global batch size:   256 | lm loss: 3.254956E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.271 | TFLOPs: 32.22 |
7:  iteration   105200/  173500 | consumed samples:     26931200 | consumed tokens:  55155097600 | elapsed time per iteration (s): 0.37 | learning rate: 8.155E-05 | global batch size:   256 | lm loss: 3.254482E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.758 | TFLOPs: 32.29 |
7:  iteration   105300/  173500 | consumed samples:     26956800 | consumed tokens:  55207526400 | elapsed time per iteration (s): 0.37 | learning rate: 8.140E-05 | global batch size:   256 | lm loss: 3.256781E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.018 | TFLOPs: 32.30 |
7:  iteration   105400/  173500 | consumed samples:     26982400 | consumed tokens:  55259955200 | elapsed time per iteration (s): 0.37 | learning rate: 8.124E-05 | global batch size:   256 | lm loss: 3.256446E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.455 | TFLOPs: 32.32 |
7:  iteration   105500/  173500 | consumed samples:     27008000 | consumed tokens:  55312384000 | elapsed time per iteration (s): 0.37 | learning rate: 8.109E-05 | global batch size:   256 | lm loss: 3.253315E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.962 | TFLOPs: 32.44 |
7:  iteration   105600/  173500 | consumed samples:     27033600 | consumed tokens:  55364812800 | elapsed time per iteration (s): 0.37 | learning rate: 8.093E-05 | global batch size:   256 | lm loss: 3.257193E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.714 | TFLOPs: 32.43 |
7:  iteration   105700/  173500 | consumed samples:     27059200 | consumed tokens:  55417241600 | elapsed time per iteration (s): 0.37 | learning rate: 8.077E-05 | global batch size:   256 | lm loss: 3.257849E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.797 | TFLOPs: 32.43 |
7:  iteration   105800/  173500 | consumed samples:     27084800 | consumed tokens:  55469670400 | elapsed time per iteration (s): 0.37 | learning rate: 8.062E-05 | global batch size:   256 | lm loss: 3.256460E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.242 | TFLOPs: 32.45 |
7:  iteration   105900/  173500 | consumed samples:     27110400 | consumed tokens:  55522099200 | elapsed time per iteration (s): 0.37 | learning rate: 8.046E-05 | global batch size:   256 | lm loss: 3.251376E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.068 | TFLOPs: 32.44 |
0: [2023-03-17 10:19:21,468] [INFO] [logging.py:68:log_dist] [Rank 0] step=106000, skipped=0, lr=[8.030787777917086e-05, 8.030787777917086e-05, 8.030787777917086e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   106000/  173500 | consumed samples:     27136000 | consumed tokens:  55574528000 | elapsed time per iteration (s): 0.37 | learning rate: 8.031E-05 | global batch size:   256 | lm loss: 3.256705E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.052 | TFLOPs: 32.44 |
0: steps: 106000 loss: 3.2761 iter time (s): 0.367 samples/sec: 697.961
7:  iteration   106100/  173500 | consumed samples:     27161600 | consumed tokens:  55626956800 | elapsed time per iteration (s): 0.37 | learning rate: 8.015E-05 | global batch size:   256 | lm loss: 3.257018E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.233 | TFLOPs: 32.45 |
7:  iteration   106200/  173500 | consumed samples:     27187200 | consumed tokens:  55679385600 | elapsed time per iteration (s): 0.37 | learning rate: 8.000E-05 | global batch size:   256 | lm loss: 3.254987E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.574 | TFLOPs: 32.47 |
7:  iteration   106300/  173500 | consumed samples:     27212800 | consumed tokens:  55731814400 | elapsed time per iteration (s): 0.37 | learning rate: 7.984E-05 | global batch size:   256 | lm loss: 3.253707E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.649 | TFLOPs: 32.42 |
7:  iteration   106400/  173500 | consumed samples:     27238400 | consumed tokens:  55784243200 | elapsed time per iteration (s): 0.37 | learning rate: 7.969E-05 | global batch size:   256 | lm loss: 3.253940E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.523 | TFLOPs: 32.46 |
7:  iteration   106500/  173500 | consumed samples:     27264000 | consumed tokens:  55836672000 | elapsed time per iteration (s): 0.37 | learning rate: 7.953E-05 | global batch size:   256 | lm loss: 3.255675E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.297 | TFLOPs: 32.45 |
7:  iteration   106600/  173500 | consumed samples:     27289600 | consumed tokens:  55889100800 | elapsed time per iteration (s): 0.37 | learning rate: 7.938E-05 | global batch size:   256 | lm loss: 3.258167E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.810 | TFLOPs: 32.43 |
7:  iteration   106700/  173500 | consumed samples:     27315200 | consumed tokens:  55941529600 | elapsed time per iteration (s): 0.37 | learning rate: 7.922E-05 | global batch size:   256 | lm loss: 3.252196E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.320 | TFLOPs: 32.27 |
7:  iteration   106800/  173500 | consumed samples:     27340800 | consumed tokens:  55993958400 | elapsed time per iteration (s): 0.37 | learning rate: 7.907E-05 | global batch size:   256 | lm loss: 3.259895E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.418 | TFLOPs: 32.37 |
7:  iteration   106900/  173500 | consumed samples:     27366400 | consumed tokens:  56046387200 | elapsed time per iteration (s): 0.37 | learning rate: 7.891E-05 | global batch size:   256 | lm loss: 3.260811E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.956 | TFLOPs: 32.34 |
7:  iteration   107000/  173500 | consumed samples:     27392000 | consumed tokens:  56098816000 | elapsed time per iteration (s): 0.37 | learning rate: 7.876E-05 | global batch size:   256 | lm loss: 3.253102E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.913 | TFLOPs: 32.44 |
7:  iteration   107100/  173500 | consumed samples:     27417600 | consumed tokens:  56151244800 | elapsed time per iteration (s): 0.37 | learning rate: 7.860E-05 | global batch size:   256 | lm loss: 3.257740E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.920 | TFLOPs: 32.44 |
7:  iteration   107200/  173500 | consumed samples:     27443200 | consumed tokens:  56203673600 | elapsed time per iteration (s): 0.37 | learning rate: 7.845E-05 | global batch size:   256 | lm loss: 3.258029E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.361 | TFLOPs: 32.46 |
7:  iteration   107300/  173500 | consumed samples:     27468800 | consumed tokens:  56256102400 | elapsed time per iteration (s): 0.37 | learning rate: 7.830E-05 | global batch size:   256 | lm loss: 3.249462E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.369 | TFLOPs: 32.41 |
7:  iteration   107400/  173500 | consumed samples:     27494400 | consumed tokens:  56308531200 | elapsed time per iteration (s): 0.37 | learning rate: 7.814E-05 | global batch size:   256 | lm loss: 3.251271E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.922 | TFLOPs: 32.44 |
7:  iteration   107500/  173500 | consumed samples:     27520000 | consumed tokens:  56360960000 | elapsed time per iteration (s): 0.37 | learning rate: 7.799E-05 | global batch size:   256 | lm loss: 3.255164E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.144 | TFLOPs: 32.40 |
7:  iteration   107600/  173500 | consumed samples:     27545600 | consumed tokens:  56413388800 | elapsed time per iteration (s): 0.37 | learning rate: 7.783E-05 | global batch size:   256 | lm loss: 3.254384E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.808 | TFLOPs: 32.43 |
7:  iteration   107700/  173500 | consumed samples:     27571200 | consumed tokens:  56465817600 | elapsed time per iteration (s): 0.37 | learning rate: 7.768E-05 | global batch size:   256 | lm loss: 3.252581E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.328 | TFLOPs: 32.32 |
7:  iteration   107800/  173500 | consumed samples:     27596800 | consumed tokens:  56518246400 | elapsed time per iteration (s): 0.37 | learning rate: 7.753E-05 | global batch size:   256 | lm loss: 3.252158E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.610 | TFLOPs: 32.24 |
7:  iteration   107900/  173500 | consumed samples:     27622400 | consumed tokens:  56570675200 | elapsed time per iteration (s): 0.37 | learning rate: 7.737E-05 | global batch size:   256 | lm loss: 3.254303E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.505 | TFLOPs: 32.23 |
0: [2023-03-17 10:31:39,324] [INFO] [logging.py:68:log_dist] [Rank 0] step=108000, skipped=0, lr=[7.722055869362951e-05, 7.722055869362951e-05, 7.722055869362951e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   108000/  173500 | consumed samples:     27648000 | consumed tokens:  56623104000 | elapsed time per iteration (s): 0.37 | learning rate: 7.722E-05 | global batch size:   256 | lm loss: 3.250675E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.450 | TFLOPs: 32.32 |
0: steps: 108000 loss: 3.2523 iter time (s): 0.367 samples/sec: 697.397
7:  iteration   108100/  173500 | consumed samples:     27673600 | consumed tokens:  56675532800 | elapsed time per iteration (s): 0.37 | learning rate: 7.707E-05 | global batch size:   256 | lm loss: 3.253465E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.729 | TFLOPs: 32.29 |
7:  iteration   108200/  173500 | consumed samples:     27699200 | consumed tokens:  56727961600 | elapsed time per iteration (s): 0.37 | learning rate: 7.691E-05 | global batch size:   256 | lm loss: 3.252955E+00 | grad norm: 0.233 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.148 | TFLOPs: 32.45 |
7:  iteration   108300/  173500 | consumed samples:     27724800 | consumed tokens:  56780390400 | elapsed time per iteration (s): 0.37 | learning rate: 7.676E-05 | global batch size:   256 | lm loss: 3.257623E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.037 | TFLOPs: 32.35 |
7:  iteration   108400/  173500 | consumed samples:     27750400 | consumed tokens:  56832819200 | elapsed time per iteration (s): 0.37 | learning rate: 7.661E-05 | global batch size:   256 | lm loss: 3.254260E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.792 | TFLOPs: 32.38 |
7:  iteration   108500/  173500 | consumed samples:     27776000 | consumed tokens:  56885248000 | elapsed time per iteration (s): 0.37 | learning rate: 7.646E-05 | global batch size:   256 | lm loss: 3.253671E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.879 | TFLOPs: 32.29 |
7:  iteration   108600/  173500 | consumed samples:     27801600 | consumed tokens:  56937676800 | elapsed time per iteration (s): 0.37 | learning rate: 7.630E-05 | global batch size:   256 | lm loss: 3.254013E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.320 | TFLOPs: 32.45 |
7:  iteration   108700/  173500 | consumed samples:     27827200 | consumed tokens:  56990105600 | elapsed time per iteration (s): 0.37 | learning rate: 7.615E-05 | global batch size:   256 | lm loss: 3.251235E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.079 | TFLOPs: 32.44 |
7:  iteration   108800/  173500 | consumed samples:     27852800 | consumed tokens:  57042534400 | elapsed time per iteration (s): 0.37 | learning rate: 7.600E-05 | global batch size:   256 | lm loss: 3.256208E+00 | grad norm: 0.249 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.233 | TFLOPs: 32.45 |
7:  iteration   108900/  173500 | consumed samples:     27878400 | consumed tokens:  57094963200 | elapsed time per iteration (s): 0.37 | learning rate: 7.585E-05 | global batch size:   256 | lm loss: 3.248484E+00 | grad norm: 0.245 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.001 | TFLOPs: 32.44 |
7:  iteration   109000/  173500 | consumed samples:     27904000 | consumed tokens:  57147392000 | elapsed time per iteration (s): 0.37 | learning rate: 7.569E-05 | global batch size:   256 | lm loss: 3.253242E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.838 | TFLOPs: 32.39 |
7:  iteration   109100/  173500 | consumed samples:     27929600 | consumed tokens:  57199820800 | elapsed time per iteration (s): 0.37 | learning rate: 7.554E-05 | global batch size:   256 | lm loss: 3.254732E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.630 | TFLOPs: 32.42 |
7:  iteration   109200/  173500 | consumed samples:     27955200 | consumed tokens:  57252249600 | elapsed time per iteration (s): 0.40 | learning rate: 7.539E-05 | global batch size:   256 | lm loss: 3.252251E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 643.818 | TFLOPs: 30.05 |
7:  iteration   109300/  173500 | consumed samples:     27980800 | consumed tokens:  57304678400 | elapsed time per iteration (s): 0.39 | learning rate: 7.524E-05 | global batch size:   256 | lm loss: 3.251515E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 655.177 | TFLOPs: 30.58 |
7:  iteration   109400/  173500 | consumed samples:     28006400 | consumed tokens:  57357107200 | elapsed time per iteration (s): 0.37 | learning rate: 7.509E-05 | global batch size:   256 | lm loss: 3.249317E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.632 | TFLOPs: 32.42 |
7:  iteration   109500/  173500 | consumed samples:     28032000 | consumed tokens:  57409536000 | elapsed time per iteration (s): 0.37 | learning rate: 7.493E-05 | global batch size:   256 | lm loss: 3.254323E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.612 | TFLOPs: 32.42 |
7:  iteration   109600/  173500 | consumed samples:     28057600 | consumed tokens:  57461964800 | elapsed time per iteration (s): 0.37 | learning rate: 7.478E-05 | global batch size:   256 | lm loss: 3.252491E+00 | grad norm: 0.207 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.753 | TFLOPs: 32.38 |
7:  iteration   109700/  173500 | consumed samples:     28083200 | consumed tokens:  57514393600 | elapsed time per iteration (s): 0.37 | learning rate: 7.463E-05 | global batch size:   256 | lm loss: 3.250238E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.760 | TFLOPs: 32.29 |
7:  iteration   109800/  173500 | consumed samples:     28108800 | consumed tokens:  57566822400 | elapsed time per iteration (s): 0.37 | learning rate: 7.448E-05 | global batch size:   256 | lm loss: 3.249306E+00 | grad norm: 0.240 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.568 | TFLOPs: 32.19 |
7:  iteration   109900/  173500 | consumed samples:     28134400 | consumed tokens:  57619251200 | elapsed time per iteration (s): 0.37 | learning rate: 7.433E-05 | global batch size:   256 | lm loss: 3.253427E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.953 | TFLOPs: 32.44 |
0: [2023-03-17 10:44:02,323] [INFO] [logging.py:68:log_dist] [Rank 0] step=110000, skipped=0, lr=[7.417709678812063e-05, 7.417709678812063e-05, 7.417709678812063e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   110000/  173500 | consumed samples:     28160000 | consumed tokens:  57671680000 | elapsed time per iteration (s): 0.37 | learning rate: 7.418E-05 | global batch size:   256 | lm loss: 3.247282E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.620 | TFLOPs: 32.42 |
0: steps: 110000 loss: 3.2207 iter time (s): 0.370 samples/sec: 692.782
7: -------------------------------------------------------------------------------------------------
7: validation loss at iteration 110000 | lm loss value: 3.385623E+00 | lm loss PPL: 2.953639E+01 | 
7: -------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration  110000 to checkpoints_146m91b400m
0: [2023-03-17 10:44:02,448] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step110000 is begin to save!
0: [2023-03-17 10:44:02,452] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_01-model_00-model_states.pt...
0: [2023-03-17 10:44:02,548] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_01-model_00-model_states.pt.
0: [2023-03-17 10:44:02,548] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_03-model_00-model_states.pt...
0: [2023-03-17 10:44:02,564] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_03-model_00-model_states.pt.
0: [2023-03-17 10:44:02,564] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_04-model_00-model_states.pt...
0: [2023-03-17 10:44:02,579] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_04-model_00-model_states.pt.
0: [2023-03-17 10:44:02,579] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_05-model_00-model_states.pt...
0: [2023-03-17 10:44:02,594] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_05-model_00-model_states.pt.
0: [2023-03-17 10:44:02,594] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_06-model_00-model_states.pt...
0: [2023-03-17 10:44:02,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_06-model_00-model_states.pt.
0: [2023-03-17 10:44:02,609] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_07-model_00-model_states.pt...
0: [2023-03-17 10:44:02,624] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_07-model_00-model_states.pt.
0: [2023-03-17 10:44:02,624] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_08-model_00-model_states.pt...
0: [2023-03-17 10:44:02,639] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_08-model_00-model_states.pt.
0: [2023-03-17 10:44:02,639] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_09-model_00-model_states.pt...
0: [2023-03-17 10:44:02,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_09-model_00-model_states.pt.
0: [2023-03-17 10:44:02,654] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_10-model_00-model_states.pt...
0: [2023-03-17 10:44:02,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_10-model_00-model_states.pt.
0: [2023-03-17 10:44:02,669] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_11-model_00-model_states.pt...
0: [2023-03-17 10:44:02,684] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_11-model_00-model_states.pt.
0: [2023-03-17 10:44:02,684] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_12-model_00-model_states.pt...
0: [2023-03-17 10:44:02,699] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_12-model_00-model_states.pt.
0: [2023-03-17 10:44:02,699] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_13-model_00-model_states.pt...
0: [2023-03-17 10:44:02,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_13-model_00-model_states.pt.
0: [2023-03-17 10:44:02,714] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_14-model_00-model_states.pt...
0: [2023-03-17 10:44:02,729] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_14-model_00-model_states.pt.
0: [2023-03-17 10:44:02,729] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_15-model_00-model_states.pt...
0: [2023-03-17 10:44:02,744] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_15-model_00-model_states.pt.
0: [2023-03-17 10:44:02,744] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_16-model_00-model_states.pt...
0: [2023-03-17 10:44:02,759] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_16-model_00-model_states.pt.
0: [2023-03-17 10:44:02,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_17-model_00-model_states.pt...
0: [2023-03-17 10:44:02,774] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_17-model_00-model_states.pt.
0: [2023-03-17 10:44:02,774] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/layer_19-model_00-model_states.pt...
0: [2023-03-17 10:44:02,775] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/layer_19-model_00-model_states.pt.
0: [2023-03-17 10:44:02,776] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step110000/mp_rank_00_model_states.pt
0: [2023-03-17 10:44:02,776] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/mp_rank_00_model_states.pt...
0: [2023-03-17 10:44:02,778] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/mp_rank_00_model_states.pt.
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
2: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
6: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
5: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 10:44:02,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
0: [2023-03-17 10:44:02,829] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,829] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,829] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,831] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,831] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,831] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,831] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,831] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,831] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,834] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,834] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,834] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,836] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,836] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,837] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,837] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,838] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,838] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,839] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,839] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 10:44:02,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
6: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,840] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,840] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,843] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,843] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 10:44:02,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,843] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,843] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 10:44:02,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,845] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
7: [2023-03-17 10:44:02,845] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 10:44:02,850] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,850] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,850] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
3: [2023-03-17 10:44:02,850] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 10:44:02,851] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,851] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,851] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
5: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,832] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,832] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,833] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,833] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,835] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,835] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,836] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,836] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,836] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
2: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 10:44:02,837] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 10:44:02,837] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,852] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
4: [2023-03-17 10:44:02,852] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0: [2023-03-17 10:44:02,859] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 10:44:02,859] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step110000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
1: [2023-03-17 10:44:02,869] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step110000 is ready now!
0:   successfully saved checkpoint at iteration  110000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 427.87
7:  iteration   110100/  173500 | consumed samples:     28185600 | consumed tokens:  57724108800 | elapsed time per iteration (s): 0.37 | learning rate: 7.403E-05 | global batch size:   256 | lm loss: 3.246705E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.200 | TFLOPs: 31.94 |
7:  iteration   110200/  173500 | consumed samples:     28211200 | consumed tokens:  57776537600 | elapsed time per iteration (s): 0.37 | learning rate: 7.388E-05 | global batch size:   256 | lm loss: 3.246959E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.848 | TFLOPs: 32.43 |
7:  iteration   110300/  173500 | consumed samples:     28236800 | consumed tokens:  57828966400 | elapsed time per iteration (s): 0.37 | learning rate: 7.372E-05 | global batch size:   256 | lm loss: 3.250947E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.912 | TFLOPs: 32.44 |
7:  iteration   110400/  173500 | consumed samples:     28262400 | consumed tokens:  57881395200 | elapsed time per iteration (s): 0.37 | learning rate: 7.357E-05 | global batch size:   256 | lm loss: 3.248094E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.464 | TFLOPs: 32.42 |
7:  iteration   110500/  173500 | consumed samples:     28288000 | consumed tokens:  57933824000 | elapsed time per iteration (s): 0.37 | learning rate: 7.342E-05 | global batch size:   256 | lm loss: 3.251809E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.486 | TFLOPs: 32.42 |
7:  iteration   110600/  173500 | consumed samples:     28313600 | consumed tokens:  57986252800 | elapsed time per iteration (s): 0.37 | learning rate: 7.327E-05 | global batch size:   256 | lm loss: 3.253752E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.036 | TFLOPs: 31.93 |
7:  iteration   110700/  173500 | consumed samples:     28339200 | consumed tokens:  58038681600 | elapsed time per iteration (s): 0.37 | learning rate: 7.312E-05 | global batch size:   256 | lm loss: 3.251139E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.953 | TFLOPs: 32.30 |
7:  iteration   110800/  173500 | consumed samples:     28364800 | consumed tokens:  58091110400 | elapsed time per iteration (s): 0.37 | learning rate: 7.297E-05 | global batch size:   256 | lm loss: 3.255021E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.133 | TFLOPs: 32.03 |
7:  iteration   110900/  173500 | consumed samples:     28390400 | consumed tokens:  58143539200 | elapsed time per iteration (s): 0.38 | learning rate: 7.282E-05 | global batch size:   256 | lm loss: 3.253001E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 676.888 | TFLOPs: 31.59 |
7:  iteration   111000/  173500 | consumed samples:     28416000 | consumed tokens:  58195968000 | elapsed time per iteration (s): 0.37 | learning rate: 7.267E-05 | global batch size:   256 | lm loss: 3.251134E+00 | grad norm: 0.208 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.575 | TFLOPs: 32.05 |
7:  iteration   111100/  173500 | consumed samples:     28441600 | consumed tokens:  58248396800 | elapsed time per iteration (s): 0.38 | learning rate: 7.252E-05 | global batch size:   256 | lm loss: 3.248564E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.160 | TFLOPs: 31.84 |
7:  iteration   111200/  173500 | consumed samples:     28467200 | consumed tokens:  58300825600 | elapsed time per iteration (s): 0.38 | learning rate: 7.237E-05 | global batch size:   256 | lm loss: 3.248993E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 673.938 | TFLOPs: 31.46 |
7:  iteration   111300/  173500 | consumed samples:     28492800 | consumed tokens:  58353254400 | elapsed time per iteration (s): 0.37 | learning rate: 7.222E-05 | global batch size:   256 | lm loss: 3.250882E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.752 | TFLOPs: 32.10 |
7:  iteration   111400/  173500 | consumed samples:     28518400 | consumed tokens:  58405683200 | elapsed time per iteration (s): 0.38 | learning rate: 7.207E-05 | global batch size:   256 | lm loss: 3.244622E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 675.410 | TFLOPs: 31.53 |
7:  iteration   111500/  173500 | consumed samples:     28544000 | consumed tokens:  58458112000 | elapsed time per iteration (s): 0.37 | learning rate: 7.193E-05 | global batch size:   256 | lm loss: 3.251710E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.160 | TFLOPs: 32.17 |
7:  iteration   111600/  173500 | consumed samples:     28569600 | consumed tokens:  58510540800 | elapsed time per iteration (s): 0.37 | learning rate: 7.178E-05 | global batch size:   256 | lm loss: 3.253637E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.066 | TFLOPs: 32.16 |
7:  iteration   111700/  173500 | consumed samples:     28595200 | consumed tokens:  58562969600 | elapsed time per iteration (s): 0.37 | learning rate: 7.163E-05 | global batch size:   256 | lm loss: 3.246256E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.515 | TFLOPs: 32.32 |
7:  iteration   111800/  173500 | consumed samples:     28620800 | consumed tokens:  58615398400 | elapsed time per iteration (s): 0.37 | learning rate: 7.148E-05 | global batch size:   256 | lm loss: 3.243911E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.940 | TFLOPs: 32.39 |
7:  iteration   111900/  173500 | consumed samples:     28646400 | consumed tokens:  58667827200 | elapsed time per iteration (s): 0.37 | learning rate: 7.133E-05 | global batch size:   256 | lm loss: 3.253238E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.443 | TFLOPs: 32.41 |
0: [2023-03-17 10:56:26,517] [INFO] [logging.py:68:log_dist] [Rank 0] step=112000, skipped=0, lr=[7.118156405567987e-05, 7.118156405567987e-05, 7.118156405567987e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   112000/  173500 | consumed samples:     28672000 | consumed tokens:  58720256000 | elapsed time per iteration (s): 0.37 | learning rate: 7.118E-05 | global batch size:   256 | lm loss: 3.251021E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.274 | TFLOPs: 32.41 |
0: steps: 112000 loss: 3.2565 iter time (s): 0.370 samples/sec: 692.007
7:  iteration   112100/  173500 | consumed samples:     28697600 | consumed tokens:  58772684800 | elapsed time per iteration (s): 0.37 | learning rate: 7.103E-05 | global batch size:   256 | lm loss: 3.253459E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.863 | TFLOPs: 32.34 |
7:  iteration   112200/  173500 | consumed samples:     28723200 | consumed tokens:  58825113600 | elapsed time per iteration (s): 0.38 | learning rate: 7.088E-05 | global batch size:   256 | lm loss: 3.247882E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 669.003 | TFLOPs: 31.23 |
7:  iteration   112300/  173500 | consumed samples:     28748800 | consumed tokens:  58877542400 | elapsed time per iteration (s): 0.37 | learning rate: 7.074E-05 | global batch size:   256 | lm loss: 3.248575E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.912 | TFLOPs: 32.44 |
7:  iteration   112400/  173500 | consumed samples:     28774400 | consumed tokens:  58929971200 | elapsed time per iteration (s): 0.37 | learning rate: 7.059E-05 | global batch size:   256 | lm loss: 3.251607E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.459 | TFLOPs: 32.41 |
7:  iteration   112500/  173500 | consumed samples:     28800000 | consumed tokens:  58982400000 | elapsed time per iteration (s): 0.37 | learning rate: 7.044E-05 | global batch size:   256 | lm loss: 3.249938E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.209 | TFLOPs: 32.40 |
7:  iteration   112600/  173500 | consumed samples:     28825600 | consumed tokens:  59034828800 | elapsed time per iteration (s): 0.37 | learning rate: 7.029E-05 | global batch size:   256 | lm loss: 3.247834E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.982 | TFLOPs: 32.30 |
7:  iteration   112700/  173500 | consumed samples:     28851200 | consumed tokens:  59087257600 | elapsed time per iteration (s): 0.37 | learning rate: 7.015E-05 | global batch size:   256 | lm loss: 3.250575E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.321 | TFLOPs: 32.31 |
7:  iteration   112800/  173500 | consumed samples:     28876800 | consumed tokens:  59139686400 | elapsed time per iteration (s): 0.37 | learning rate: 7.000E-05 | global batch size:   256 | lm loss: 3.246619E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.898 | TFLOPs: 32.34 |
7:  iteration   112900/  173500 | consumed samples:     28902400 | consumed tokens:  59192115200 | elapsed time per iteration (s): 0.37 | learning rate: 6.985E-05 | global batch size:   256 | lm loss: 3.250970E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.384 | TFLOPs: 32.22 |
7:  iteration   113000/  173500 | consumed samples:     28928000 | consumed tokens:  59244544000 | elapsed time per iteration (s): 0.37 | learning rate: 6.970E-05 | global batch size:   256 | lm loss: 3.249741E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.487 | TFLOPs: 32.37 |
7:  iteration   113100/  173500 | consumed samples:     28953600 | consumed tokens:  59296972800 | elapsed time per iteration (s): 0.37 | learning rate: 6.956E-05 | global batch size:   256 | lm loss: 3.245514E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.506 | TFLOPs: 32.32 |
7:  iteration   113200/  173500 | consumed samples:     28979200 | consumed tokens:  59349401600 | elapsed time per iteration (s): 0.37 | learning rate: 6.941E-05 | global batch size:   256 | lm loss: 3.251129E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.574 | TFLOPs: 32.42 |
7:  iteration   113300/  173500 | consumed samples:     29004800 | consumed tokens:  59401830400 | elapsed time per iteration (s): 0.37 | learning rate: 6.926E-05 | global batch size:   256 | lm loss: 3.249725E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.946 | TFLOPs: 32.44 |
7:  iteration   113400/  173500 | consumed samples:     29030400 | consumed tokens:  59454259200 | elapsed time per iteration (s): 0.37 | learning rate: 6.912E-05 | global batch size:   256 | lm loss: 3.247881E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.577 | TFLOPs: 32.37 |
7:  iteration   113500/  173500 | consumed samples:     29056000 | consumed tokens:  59506688000 | elapsed time per iteration (s): 0.37 | learning rate: 6.897E-05 | global batch size:   256 | lm loss: 3.246060E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.621 | TFLOPs: 32.42 |
7:  iteration   113600/  173500 | consumed samples:     29081600 | consumed tokens:  59559116800 | elapsed time per iteration (s): 0.37 | learning rate: 6.882E-05 | global batch size:   256 | lm loss: 3.251021E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.309 | TFLOPs: 32.41 |
7:  iteration   113700/  173500 | consumed samples:     29107200 | consumed tokens:  59611545600 | elapsed time per iteration (s): 0.37 | learning rate: 6.868E-05 | global batch size:   256 | lm loss: 3.246717E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.849 | TFLOPs: 32.39 |
7:  iteration   113800/  173500 | consumed samples:     29132800 | consumed tokens:  59663974400 | elapsed time per iteration (s): 0.37 | learning rate: 6.853E-05 | global batch size:   256 | lm loss: 3.251376E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.277 | TFLOPs: 32.31 |
7:  iteration   113900/  173500 | consumed samples:     29158400 | consumed tokens:  59716403200 | elapsed time per iteration (s): 0.37 | learning rate: 6.838E-05 | global batch size:   256 | lm loss: 3.245817E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.831 | TFLOPs: 32.34 |
0: [2023-03-17 11:08:47,063] [INFO] [logging.py:68:log_dist] [Rank 0] step=114000, skipped=0, lr=[6.823796836261315e-05, 6.823796836261315e-05, 6.823796836261315e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   114000/  173500 | consumed samples:     29184000 | consumed tokens:  59768832000 | elapsed time per iteration (s): 0.38 | learning rate: 6.824E-05 | global batch size:   256 | lm loss: 3.245206E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.741 | TFLOPs: 31.68 |
0: steps: 114000 loss: 3.2378 iter time (s): 0.368 samples/sec: 695.121
7:  iteration   114100/  173500 | consumed samples:     29209600 | consumed tokens:  59821260800 | elapsed time per iteration (s): 0.37 | learning rate: 6.809E-05 | global batch size:   256 | lm loss: 3.248135E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.607 | TFLOPs: 32.42 |
7:  iteration   114200/  173500 | consumed samples:     29235200 | consumed tokens:  59873689600 | elapsed time per iteration (s): 0.37 | learning rate: 6.795E-05 | global batch size:   256 | lm loss: 3.248278E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.693 | TFLOPs: 32.43 |
7:  iteration   114300/  173500 | consumed samples:     29260800 | consumed tokens:  59926118400 | elapsed time per iteration (s): 0.37 | learning rate: 6.780E-05 | global batch size:   256 | lm loss: 3.251901E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.145 | TFLOPs: 32.45 |
7:  iteration   114400/  173500 | consumed samples:     29286400 | consumed tokens:  59978547200 | elapsed time per iteration (s): 0.37 | learning rate: 6.766E-05 | global batch size:   256 | lm loss: 3.244405E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.151 | TFLOPs: 32.45 |
7:  iteration   114500/  173500 | consumed samples:     29312000 | consumed tokens:  60030976000 | elapsed time per iteration (s): 0.37 | learning rate: 6.751E-05 | global batch size:   256 | lm loss: 3.249789E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.033 | TFLOPs: 32.44 |
7:  iteration   114600/  173500 | consumed samples:     29337600 | consumed tokens:  60083404800 | elapsed time per iteration (s): 0.37 | learning rate: 6.737E-05 | global batch size:   256 | lm loss: 3.248411E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.491 | TFLOPs: 32.42 |
7:  iteration   114700/  173500 | consumed samples:     29363200 | consumed tokens:  60135833600 | elapsed time per iteration (s): 0.37 | learning rate: 6.722E-05 | global batch size:   256 | lm loss: 3.247874E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.232 | TFLOPs: 32.45 |
7:  iteration   114800/  173500 | consumed samples:     29388800 | consumed tokens:  60188262400 | elapsed time per iteration (s): 0.37 | learning rate: 6.708E-05 | global batch size:   256 | lm loss: 3.247401E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.029 | TFLOPs: 32.44 |
7:  iteration   114900/  173500 | consumed samples:     29414400 | consumed tokens:  60240691200 | elapsed time per iteration (s): 0.37 | learning rate: 6.693E-05 | global batch size:   256 | lm loss: 3.249336E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.657 | TFLOPs: 32.42 |
7:  iteration   115000/  173500 | consumed samples:     29440000 | consumed tokens:  60293120000 | elapsed time per iteration (s): 0.37 | learning rate: 6.679E-05 | global batch size:   256 | lm loss: 3.251220E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.924 | TFLOPs: 32.39 |
7:  iteration   115100/  173500 | consumed samples:     29465600 | consumed tokens:  60345548800 | elapsed time per iteration (s): 0.37 | learning rate: 6.664E-05 | global batch size:   256 | lm loss: 3.244974E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.194 | TFLOPs: 32.40 |
7:  iteration   115200/  173500 | consumed samples:     29491200 | consumed tokens:  60397977600 | elapsed time per iteration (s): 0.37 | learning rate: 6.650E-05 | global batch size:   256 | lm loss: 3.247116E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.930 | TFLOPs: 32.44 |
7:  iteration   115300/  173500 | consumed samples:     29516800 | consumed tokens:  60450406400 | elapsed time per iteration (s): 0.37 | learning rate: 6.635E-05 | global batch size:   256 | lm loss: 3.246726E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.506 | TFLOPs: 32.42 |
7:  iteration   115400/  173500 | consumed samples:     29542400 | consumed tokens:  60502835200 | elapsed time per iteration (s): 0.37 | learning rate: 6.621E-05 | global batch size:   256 | lm loss: 3.248189E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.896 | TFLOPs: 32.44 |
7:  iteration   115500/  173500 | consumed samples:     29568000 | consumed tokens:  60555264000 | elapsed time per iteration (s): 0.37 | learning rate: 6.607E-05 | global batch size:   256 | lm loss: 3.250743E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.160 | TFLOPs: 32.45 |
7:  iteration   115600/  173500 | consumed samples:     29593600 | consumed tokens:  60607692800 | elapsed time per iteration (s): 0.37 | learning rate: 6.592E-05 | global batch size:   256 | lm loss: 3.247448E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.913 | TFLOPs: 31.97 |
7:  iteration   115700/  173500 | consumed samples:     29619200 | consumed tokens:  60660121600 | elapsed time per iteration (s): 0.37 | learning rate: 6.578E-05 | global batch size:   256 | lm loss: 3.248500E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.682 | TFLOPs: 32.10 |
7:  iteration   115800/  173500 | consumed samples:     29644800 | consumed tokens:  60712550400 | elapsed time per iteration (s): 0.37 | learning rate: 6.564E-05 | global batch size:   256 | lm loss: 3.245475E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.454 | TFLOPs: 32.37 |
7:  iteration   115900/  173500 | consumed samples:     29670400 | consumed tokens:  60764979200 | elapsed time per iteration (s): 0.37 | learning rate: 6.549E-05 | global batch size:   256 | lm loss: 3.247927E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.861 | TFLOPs: 32.25 |
0: [2023-03-17 11:21:05,182] [INFO] [logging.py:68:log_dist] [Rank 0] step=116000, skipped=0, lr=[6.535024808618106e-05, 6.535024808618106e-05, 6.535024808618106e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   116000/  173500 | consumed samples:     29696000 | consumed tokens:  60817408000 | elapsed time per iteration (s): 0.37 | learning rate: 6.535E-05 | global batch size:   256 | lm loss: 3.246090E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.777 | TFLOPs: 32.43 |
0: steps: 116000 loss: 3.2578 iter time (s): 0.367 samples/sec: 697.396
7:  iteration   116100/  173500 | consumed samples:     29721600 | consumed tokens:  60869836800 | elapsed time per iteration (s): 0.37 | learning rate: 6.521E-05 | global batch size:   256 | lm loss: 3.244227E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.061 | TFLOPs: 32.44 |
7:  iteration   116200/  173500 | consumed samples:     29747200 | consumed tokens:  60922265600 | elapsed time per iteration (s): 0.37 | learning rate: 6.506E-05 | global batch size:   256 | lm loss: 3.246706E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.326 | TFLOPs: 32.46 |
7:  iteration   116300/  173500 | consumed samples:     29772800 | consumed tokens:  60974694400 | elapsed time per iteration (s): 0.37 | learning rate: 6.492E-05 | global batch size:   256 | lm loss: 3.247682E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.387 | TFLOPs: 32.41 |
7:  iteration   116400/  173500 | consumed samples:     29798400 | consumed tokens:  61027123200 | elapsed time per iteration (s): 0.37 | learning rate: 6.478E-05 | global batch size:   256 | lm loss: 3.244458E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.984 | TFLOPs: 32.39 |
7:  iteration   116500/  173500 | consumed samples:     29824000 | consumed tokens:  61079552000 | elapsed time per iteration (s): 0.37 | learning rate: 6.464E-05 | global batch size:   256 | lm loss: 3.241545E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.507 | TFLOPs: 32.46 |
7:  iteration   116600/  173500 | consumed samples:     29849600 | consumed tokens:  61131980800 | elapsed time per iteration (s): 0.37 | learning rate: 6.450E-05 | global batch size:   256 | lm loss: 3.242759E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.296 | TFLOPs: 32.45 |
7:  iteration   116700/  173500 | consumed samples:     29875200 | consumed tokens:  61184409600 | elapsed time per iteration (s): 0.37 | learning rate: 6.435E-05 | global batch size:   256 | lm loss: 3.249590E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.422 | TFLOPs: 32.46 |
7:  iteration   116800/  173500 | consumed samples:     29900800 | consumed tokens:  61236838400 | elapsed time per iteration (s): 0.37 | learning rate: 6.421E-05 | global batch size:   256 | lm loss: 3.248362E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.367 | TFLOPs: 32.46 |
7:  iteration   116900/  173500 | consumed samples:     29926400 | consumed tokens:  61289267200 | elapsed time per iteration (s): 0.37 | learning rate: 6.407E-05 | global batch size:   256 | lm loss: 3.244346E+00 | grad norm: 0.241 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.423 | TFLOPs: 32.37 |
7:  iteration   117000/  173500 | consumed samples:     29952000 | consumed tokens:  61341696000 | elapsed time per iteration (s): 0.37 | learning rate: 6.393E-05 | global batch size:   256 | lm loss: 3.248662E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.907 | TFLOPs: 32.44 |
7:  iteration   117100/  173500 | consumed samples:     29977600 | consumed tokens:  61394124800 | elapsed time per iteration (s): 0.37 | learning rate: 6.379E-05 | global batch size:   256 | lm loss: 3.244107E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.359 | TFLOPs: 32.41 |
7:  iteration   117200/  173500 | consumed samples:     30003200 | consumed tokens:  61446553600 | elapsed time per iteration (s): 0.37 | learning rate: 6.365E-05 | global batch size:   256 | lm loss: 3.245619E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.819 | TFLOPs: 32.15 |
7:  iteration   117300/  173500 | consumed samples:     30028800 | consumed tokens:  61498982400 | elapsed time per iteration (s): 0.39 | learning rate: 6.351E-05 | global batch size:   256 | lm loss: 3.246160E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 659.648 | TFLOPs: 30.79 |
7:  iteration   117400/  173500 | consumed samples:     30054400 | consumed tokens:  61551411200 | elapsed time per iteration (s): 0.39 | learning rate: 6.336E-05 | global batch size:   256 | lm loss: 3.248775E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 657.805 | TFLOPs: 30.70 |
7:  iteration   117500/  173500 | consumed samples:     30080000 | consumed tokens:  61603840000 | elapsed time per iteration (s): 0.37 | learning rate: 6.322E-05 | global batch size:   256 | lm loss: 3.243969E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.802 | TFLOPs: 32.48 |
7:  iteration   117600/  173500 | consumed samples:     30105600 | consumed tokens:  61656268800 | elapsed time per iteration (s): 0.37 | learning rate: 6.308E-05 | global batch size:   256 | lm loss: 3.244988E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.965 | TFLOPs: 32.49 |
7:  iteration   117700/  173500 | consumed samples:     30131200 | consumed tokens:  61708697600 | elapsed time per iteration (s): 0.37 | learning rate: 6.294E-05 | global batch size:   256 | lm loss: 3.240757E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.807 | TFLOPs: 32.48 |
7:  iteration   117800/  173500 | consumed samples:     30156800 | consumed tokens:  61761126400 | elapsed time per iteration (s): 0.37 | learning rate: 6.280E-05 | global batch size:   256 | lm loss: 3.244203E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.680 | TFLOPs: 32.47 |
7:  iteration   117900/  173500 | consumed samples:     30182400 | consumed tokens:  61813555200 | elapsed time per iteration (s): 0.37 | learning rate: 6.266E-05 | global batch size:   256 | lm loss: 3.244507E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.795 | TFLOPs: 32.48 |
0: [2023-03-17 11:33:26,118] [INFO] [logging.py:68:log_dist] [Rank 0] step=118000, skipped=0, lr=[6.252226684525562e-05, 6.252226684525562e-05, 6.252226684525562e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   118000/  173500 | consumed samples:     30208000 | consumed tokens:  61865984000 | elapsed time per iteration (s): 0.37 | learning rate: 6.252E-05 | global batch size:   256 | lm loss: 3.246761E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.601 | TFLOPs: 32.47 |
0: steps: 118000 loss: 3.2702 iter time (s): 0.369 samples/sec: 694.592
7:  iteration   118100/  173500 | consumed samples:     30233600 | consumed tokens:  61918412800 | elapsed time per iteration (s): 0.37 | learning rate: 6.238E-05 | global batch size:   256 | lm loss: 3.246908E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.283 | TFLOPs: 32.45 |
7:  iteration   118200/  173500 | consumed samples:     30259200 | consumed tokens:  61970841600 | elapsed time per iteration (s): 0.48 | learning rate: 6.224E-05 | global batch size:   256 | lm loss: 3.247471E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 528.647 | TFLOPs: 24.68 |
7:  iteration   118300/  173500 | consumed samples:     30284800 | consumed tokens:  62023270400 | elapsed time per iteration (s): 0.37 | learning rate: 6.210E-05 | global batch size:   256 | lm loss: 3.244902E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.009 | TFLOPs: 32.35 |
7:  iteration   118400/  173500 | consumed samples:     30310400 | consumed tokens:  62075699200 | elapsed time per iteration (s): 0.37 | learning rate: 6.196E-05 | global batch size:   256 | lm loss: 3.244463E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.610 | TFLOPs: 32.38 |
7:  iteration   118500/  173500 | consumed samples:     30336000 | consumed tokens:  62128128000 | elapsed time per iteration (s): 0.37 | learning rate: 6.183E-05 | global batch size:   256 | lm loss: 3.248090E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.594 | TFLOPs: 32.47 |
7:  iteration   118600/  173500 | consumed samples:     30361600 | consumed tokens:  62180556800 | elapsed time per iteration (s): 0.37 | learning rate: 6.169E-05 | global batch size:   256 | lm loss: 3.245304E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.749 | TFLOPs: 32.33 |
7:  iteration   118700/  173500 | consumed samples:     30387200 | consumed tokens:  62232985600 | elapsed time per iteration (s): 0.37 | learning rate: 6.155E-05 | global batch size:   256 | lm loss: 3.246457E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.453 | TFLOPs: 32.32 |
7:  iteration   118800/  173500 | consumed samples:     30412800 | consumed tokens:  62285414400 | elapsed time per iteration (s): 0.37 | learning rate: 6.141E-05 | global batch size:   256 | lm loss: 3.240005E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.948 | TFLOPs: 32.44 |
7:  iteration   118900/  173500 | consumed samples:     30438400 | consumed tokens:  62337843200 | elapsed time per iteration (s): 0.37 | learning rate: 6.127E-05 | global batch size:   256 | lm loss: 3.249087E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.582 | TFLOPs: 32.37 |
7:  iteration   119000/  173500 | consumed samples:     30464000 | consumed tokens:  62390272000 | elapsed time per iteration (s): 0.37 | learning rate: 6.113E-05 | global batch size:   256 | lm loss: 3.245195E+00 | grad norm: 0.206 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.858 | TFLOPs: 32.29 |
7:  iteration   119100/  173500 | consumed samples:     30489600 | consumed tokens:  62442700800 | elapsed time per iteration (s): 0.37 | learning rate: 6.099E-05 | global batch size:   256 | lm loss: 3.245020E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.750 | TFLOPs: 32.43 |
7:  iteration   119200/  173500 | consumed samples:     30515200 | consumed tokens:  62495129600 | elapsed time per iteration (s): 0.37 | learning rate: 6.086E-05 | global batch size:   256 | lm loss: 3.247274E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.773 | TFLOPs: 32.34 |
7:  iteration   119300/  173500 | consumed samples:     30540800 | consumed tokens:  62547558400 | elapsed time per iteration (s): 0.37 | learning rate: 6.072E-05 | global batch size:   256 | lm loss: 3.242739E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.969 | TFLOPs: 32.44 |
7:  iteration   119400/  173500 | consumed samples:     30566400 | consumed tokens:  62599987200 | elapsed time per iteration (s): 0.37 | learning rate: 6.058E-05 | global batch size:   256 | lm loss: 3.239743E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.489 | TFLOPs: 32.46 |
7:  iteration   119500/  173500 | consumed samples:     30592000 | consumed tokens:  62652416000 | elapsed time per iteration (s): 0.37 | learning rate: 6.044E-05 | global batch size:   256 | lm loss: 3.240617E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.463 | TFLOPs: 32.41 |
7:  iteration   119600/  173500 | consumed samples:     30617600 | consumed tokens:  62704844800 | elapsed time per iteration (s): 0.37 | learning rate: 6.031E-05 | global batch size:   256 | lm loss: 3.241267E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.604 | TFLOPs: 32.47 |
7:  iteration   119700/  173500 | consumed samples:     30643200 | consumed tokens:  62757273600 | elapsed time per iteration (s): 0.37 | learning rate: 6.017E-05 | global batch size:   256 | lm loss: 3.247594E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.544 | TFLOPs: 32.47 |
7:  iteration   119800/  173500 | consumed samples:     30668800 | consumed tokens:  62809702400 | elapsed time per iteration (s): 0.38 | learning rate: 6.003E-05 | global batch size:   256 | lm loss: 3.243846E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 669.721 | TFLOPs: 31.26 |
7:  iteration   119900/  173500 | consumed samples:     30694400 | consumed tokens:  62862131200 | elapsed time per iteration (s): 0.37 | learning rate: 5.989E-05 | global batch size:   256 | lm loss: 3.243072E+00 | grad norm: 0.236 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.103 | TFLOPs: 32.44 |
0: [2023-03-17 11:45:56,476] [INFO] [logging.py:68:log_dist] [Rank 0] step=120000, skipped=0, lr=[5.975780833100023e-05, 5.975780833100023e-05, 5.975780833100023e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   120000/  173500 | consumed samples:     30720000 | consumed tokens:  62914560000 | elapsed time per iteration (s): 0.37 | learning rate: 5.976E-05 | global batch size:   256 | lm loss: 3.241110E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.352 | TFLOPs: 32.46 |
0: steps: 120000 loss: 3.1865 iter time (s): 0.373 samples/sec: 686.132
7: -------------------------------------------------------------------------------------------------
7: validation loss at iteration 120000 | lm loss value: 3.267115E+00 | lm loss PPL: 2.623553E+01 | 
7: -------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration  120000 to checkpoints_146m91b400m
0: [2023-03-17 11:45:56,620] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step120000 is begin to save!
0: [2023-03-17 11:45:56,625] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_01-model_00-model_states.pt...
0: [2023-03-17 11:45:56,730] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_01-model_00-model_states.pt.
0: [2023-03-17 11:45:56,731] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_03-model_00-model_states.pt...
0: [2023-03-17 11:45:56,747] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_03-model_00-model_states.pt.
0: [2023-03-17 11:45:56,747] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_04-model_00-model_states.pt...
0: [2023-03-17 11:45:56,762] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_04-model_00-model_states.pt.
0: [2023-03-17 11:45:56,762] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_05-model_00-model_states.pt...
0: [2023-03-17 11:45:56,777] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_05-model_00-model_states.pt.
0: [2023-03-17 11:45:56,778] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_06-model_00-model_states.pt...
0: [2023-03-17 11:45:56,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_06-model_00-model_states.pt.
0: [2023-03-17 11:45:56,793] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_07-model_00-model_states.pt...
0: [2023-03-17 11:45:56,808] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_07-model_00-model_states.pt.
0: [2023-03-17 11:45:56,808] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_08-model_00-model_states.pt...
0: [2023-03-17 11:45:56,823] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_08-model_00-model_states.pt.
0: [2023-03-17 11:45:56,823] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_09-model_00-model_states.pt...
0: [2023-03-17 11:45:56,838] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_09-model_00-model_states.pt.
0: [2023-03-17 11:45:56,838] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_10-model_00-model_states.pt...
0: [2023-03-17 11:45:56,853] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_10-model_00-model_states.pt.
0: [2023-03-17 11:45:56,854] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_11-model_00-model_states.pt...
0: [2023-03-17 11:45:56,869] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_11-model_00-model_states.pt.
0: [2023-03-17 11:45:56,869] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_12-model_00-model_states.pt...
0: [2023-03-17 11:45:56,884] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_12-model_00-model_states.pt.
0: [2023-03-17 11:45:56,884] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_13-model_00-model_states.pt...
0: [2023-03-17 11:45:56,899] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_13-model_00-model_states.pt.
0: [2023-03-17 11:45:56,899] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_14-model_00-model_states.pt...
0: [2023-03-17 11:45:56,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_14-model_00-model_states.pt.
0: [2023-03-17 11:45:56,915] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_15-model_00-model_states.pt...
0: [2023-03-17 11:45:56,930] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_15-model_00-model_states.pt.
0: [2023-03-17 11:45:56,930] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_16-model_00-model_states.pt...
0: [2023-03-17 11:45:56,945] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_16-model_00-model_states.pt.
0: [2023-03-17 11:45:56,945] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_17-model_00-model_states.pt...
0: [2023-03-17 11:45:56,960] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_17-model_00-model_states.pt.
0: [2023-03-17 11:45:56,960] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/layer_19-model_00-model_states.pt...
0: [2023-03-17 11:45:56,961] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/layer_19-model_00-model_states.pt.
0: [2023-03-17 11:45:56,962] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step120000/mp_rank_00_model_states.pt
0: [2023-03-17 11:45:56,962] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/mp_rank_00_model_states.pt...
0: [2023-03-17 11:45:56,966] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/mp_rank_00_model_states.pt.
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
0: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
5: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
1: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
7: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
6: [2023-03-17 11:45:56,983] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,019] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,019] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,019] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,019] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,019] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,020] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,020] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,020] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,020] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,020] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,020] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,020] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,022] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,022] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
3: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
3: [2023-03-17 11:45:57,023] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,023] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
3: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,023] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,024] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,024] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,024] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,024] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,024] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,029] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,029] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,029] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,029] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,029] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,032] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,034] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,034] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,034] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,033] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,033] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,033] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,033] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,033] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,033] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,035] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,035] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,035] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,035] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 11:45:57,037] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,037] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,037] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
2: [2023-03-17 11:45:57,037] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,042] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,043] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,043] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,043] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,044] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,044] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,044] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,044] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,045] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,045] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,045] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,045] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
7: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 11:45:57,045] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 11:45:57,045] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,046] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,046] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,046] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,046] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,046] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,047] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,047] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,047] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,047] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,047] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,048] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,048] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 11:45:57,048] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,048] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 11:45:57,048] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
6: [2023-03-17 11:45:57,048] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0: [2023-03-17 11:45:57,049] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 11:45:57,049] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
5: [2023-03-17 11:45:57,050] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 11:45:57,050] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 11:45:57,050] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,052] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,052] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,052] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,052] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
4: [2023-03-17 11:45:57,053] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 11:45:57,053] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step120000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,061] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,062] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,062] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
1: [2023-03-17 11:45:57,062] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step120000 is ready now!
0:   successfully saved checkpoint at iteration  120000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 467.10
7:  iteration   120100/  173500 | consumed samples:     30745600 | consumed tokens:  62966988800 | elapsed time per iteration (s): 0.37 | learning rate: 5.962E-05 | global batch size:   256 | lm loss: 3.245476E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.054 | TFLOPs: 31.93 |
7:  iteration   120200/  173500 | consumed samples:     30771200 | consumed tokens:  63019417600 | elapsed time per iteration (s): 0.39 | learning rate: 5.948E-05 | global batch size:   256 | lm loss: 3.241960E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 664.450 | TFLOPs: 31.01 |
7:  iteration   120300/  173500 | consumed samples:     30796800 | consumed tokens:  63071846400 | elapsed time per iteration (s): 0.37 | learning rate: 5.935E-05 | global batch size:   256 | lm loss: 3.244433E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.056 | TFLOPs: 32.44 |
7:  iteration   120400/  173500 | consumed samples:     30822400 | consumed tokens:  63124275200 | elapsed time per iteration (s): 0.37 | learning rate: 5.921E-05 | global batch size:   256 | lm loss: 3.242888E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.969 | TFLOPs: 32.02 |
7:  iteration   120500/  173500 | consumed samples:     30848000 | consumed tokens:  63176704000 | elapsed time per iteration (s): 0.37 | learning rate: 5.908E-05 | global batch size:   256 | lm loss: 3.244461E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.813 | TFLOPs: 32.01 |
7:  iteration   120600/  173500 | consumed samples:     30873600 | consumed tokens:  63229132800 | elapsed time per iteration (s): 0.37 | learning rate: 5.894E-05 | global batch size:   256 | lm loss: 3.242574E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.488 | TFLOPs: 32.28 |
7:  iteration   120700/  173500 | consumed samples:     30899200 | consumed tokens:  63281561600 | elapsed time per iteration (s): 0.37 | learning rate: 5.881E-05 | global batch size:   256 | lm loss: 3.246718E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.443 | TFLOPs: 32.23 |
7:  iteration   120800/  173500 | consumed samples:     30924800 | consumed tokens:  63333990400 | elapsed time per iteration (s): 0.37 | learning rate: 5.867E-05 | global batch size:   256 | lm loss: 3.242693E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.306 | TFLOPs: 32.22 |
7:  iteration   120900/  173500 | consumed samples:     30950400 | consumed tokens:  63386419200 | elapsed time per iteration (s): 0.37 | learning rate: 5.854E-05 | global batch size:   256 | lm loss: 3.239696E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.613 | TFLOPs: 32.28 |
7:  iteration   121000/  173500 | consumed samples:     30976000 | consumed tokens:  63438848000 | elapsed time per iteration (s): 0.37 | learning rate: 5.840E-05 | global batch size:   256 | lm loss: 3.243532E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.702 | TFLOPs: 32.33 |
7:  iteration   121100/  173500 | consumed samples:     31001600 | consumed tokens:  63491276800 | elapsed time per iteration (s): 0.37 | learning rate: 5.827E-05 | global batch size:   256 | lm loss: 3.243558E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.496 | TFLOPs: 31.90 |
7:  iteration   121200/  173500 | consumed samples:     31027200 | consumed tokens:  63543705600 | elapsed time per iteration (s): 0.37 | learning rate: 5.813E-05 | global batch size:   256 | lm loss: 3.243864E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.938 | TFLOPs: 32.16 |
7:  iteration   121300/  173500 | consumed samples:     31052800 | consumed tokens:  63596134400 | elapsed time per iteration (s): 0.37 | learning rate: 5.800E-05 | global batch size:   256 | lm loss: 3.242402E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.541 | TFLOPs: 32.09 |
7:  iteration   121400/  173500 | consumed samples:     31078400 | consumed tokens:  63648563200 | elapsed time per iteration (s): 0.37 | learning rate: 5.786E-05 | global batch size:   256 | lm loss: 3.246299E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.956 | TFLOPs: 32.30 |
7:  iteration   121500/  173500 | consumed samples:     31104000 | consumed tokens:  63700992000 | elapsed time per iteration (s): 0.38 | learning rate: 5.773E-05 | global batch size:   256 | lm loss: 3.242229E+00 | grad norm: 0.237 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.117 | TFLOPs: 31.70 |
7:  iteration   121600/  173500 | consumed samples:     31129600 | consumed tokens:  63753420800 | elapsed time per iteration (s): 0.37 | learning rate: 5.759E-05 | global batch size:   256 | lm loss: 3.242755E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 686.905 | TFLOPs: 32.06 |
7:  iteration   121700/  173500 | consumed samples:     31155200 | consumed tokens:  63805849600 | elapsed time per iteration (s): 0.38 | learning rate: 5.746E-05 | global batch size:   256 | lm loss: 3.240781E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.232 | TFLOPs: 31.38 |
7:  iteration   121800/  173500 | consumed samples:     31180800 | consumed tokens:  63858278400 | elapsed time per iteration (s): 0.37 | learning rate: 5.733E-05 | global batch size:   256 | lm loss: 3.239920E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.932 | TFLOPs: 32.11 |
7:  iteration   121900/  173500 | consumed samples:     31206400 | consumed tokens:  63910707200 | elapsed time per iteration (s): 0.37 | learning rate: 5.719E-05 | global batch size:   256 | lm loss: 3.241132E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.079 | TFLOPs: 32.12 |
0: [2023-03-17 11:58:22,459] [INFO] [logging.py:68:log_dist] [Rank 0] step=122000, skipped=0, lr=[5.706057124448849e-05, 5.706057124448849e-05, 5.706057124448849e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   122000/  173500 | consumed samples:     31232000 | consumed tokens:  63963136000 | elapsed time per iteration (s): 0.37 | learning rate: 5.706E-05 | global batch size:   256 | lm loss: 3.245048E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.243 | TFLOPs: 32.22 |
0: steps: 122000 loss: 3.2304 iter time (s): 0.371 samples/sec: 689.705
7:  iteration   122100/  173500 | consumed samples:     31257600 | consumed tokens:  64015564800 | elapsed time per iteration (s): 0.37 | learning rate: 5.693E-05 | global batch size:   256 | lm loss: 3.247194E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.174 | TFLOPs: 32.26 |
7:  iteration   122200/  173500 | consumed samples:     31283200 | consumed tokens:  64067993600 | elapsed time per iteration (s): 0.37 | learning rate: 5.679E-05 | global batch size:   256 | lm loss: 3.240746E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.934 | TFLOPs: 32.11 |
7:  iteration   122300/  173500 | consumed samples:     31308800 | consumed tokens:  64120422400 | elapsed time per iteration (s): 0.37 | learning rate: 5.666E-05 | global batch size:   256 | lm loss: 3.242405E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.763 | TFLOPs: 32.29 |
7:  iteration   122400/  173500 | consumed samples:     31334400 | consumed tokens:  64172851200 | elapsed time per iteration (s): 0.37 | learning rate: 5.653E-05 | global batch size:   256 | lm loss: 3.243281E+00 | grad norm: 0.211 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.542 | TFLOPs: 32.23 |
7:  iteration   122500/  173500 | consumed samples:     31360000 | consumed tokens:  64225280000 | elapsed time per iteration (s): 0.37 | learning rate: 5.640E-05 | global batch size:   256 | lm loss: 3.241102E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.955 | TFLOPs: 32.11 |
7:  iteration   122600/  173500 | consumed samples:     31385600 | consumed tokens:  64277708800 | elapsed time per iteration (s): 0.37 | learning rate: 5.627E-05 | global batch size:   256 | lm loss: 3.242083E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.778 | TFLOPs: 32.15 |
7:  iteration   122700/  173500 | consumed samples:     31411200 | consumed tokens:  64330137600 | elapsed time per iteration (s): 0.37 | learning rate: 5.613E-05 | global batch size:   256 | lm loss: 3.242836E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 684.512 | TFLOPs: 31.95 |
7:  iteration   122800/  173500 | consumed samples:     31436800 | consumed tokens:  64382566400 | elapsed time per iteration (s): 0.37 | learning rate: 5.600E-05 | global batch size:   256 | lm loss: 3.240803E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.154 | TFLOPs: 31.89 |
7:  iteration   122900/  173500 | consumed samples:     31462400 | consumed tokens:  64434995200 | elapsed time per iteration (s): 0.37 | learning rate: 5.587E-05 | global batch size:   256 | lm loss: 3.241156E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.184 | TFLOPs: 32.17 |
7:  iteration   123000/  173500 | consumed samples:     31488000 | consumed tokens:  64487424000 | elapsed time per iteration (s): 0.37 | learning rate: 5.574E-05 | global batch size:   256 | lm loss: 3.240446E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.402 | TFLOPs: 32.13 |
7:  iteration   123100/  173500 | consumed samples:     31513600 | consumed tokens:  64539852800 | elapsed time per iteration (s): 0.37 | learning rate: 5.561E-05 | global batch size:   256 | lm loss: 3.239709E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 691.228 | TFLOPs: 32.26 |
7:  iteration   123200/  173500 | consumed samples:     31539200 | consumed tokens:  64592281600 | elapsed time per iteration (s): 0.37 | learning rate: 5.548E-05 | global batch size:   256 | lm loss: 3.242625E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 685.062 | TFLOPs: 31.98 |
7:  iteration   123300/  173500 | consumed samples:     31564800 | consumed tokens:  64644710400 | elapsed time per iteration (s): 0.37 | learning rate: 5.535E-05 | global batch size:   256 | lm loss: 3.241026E+00 | grad norm: 0.242 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.858 | TFLOPs: 32.20 |
7:  iteration   123400/  173500 | consumed samples:     31590400 | consumed tokens:  64697139200 | elapsed time per iteration (s): 0.37 | learning rate: 5.521E-05 | global batch size:   256 | lm loss: 3.242721E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.138 | TFLOPs: 32.12 |
7:  iteration   123500/  173500 | consumed samples:     31616000 | consumed tokens:  64749568000 | elapsed time per iteration (s): 0.37 | learning rate: 5.508E-05 | global batch size:   256 | lm loss: 3.238554E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.100 | TFLOPs: 32.12 |
7:  iteration   123600/  173500 | consumed samples:     31641600 | consumed tokens:  64801996800 | elapsed time per iteration (s): 0.37 | learning rate: 5.495E-05 | global batch size:   256 | lm loss: 3.240772E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.289 | TFLOPs: 32.08 |
7:  iteration   123700/  173500 | consumed samples:     31667200 | consumed tokens:  64854425600 | elapsed time per iteration (s): 0.37 | learning rate: 5.482E-05 | global batch size:   256 | lm loss: 3.244301E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.621 | TFLOPs: 32.14 |
7:  iteration   123800/  173500 | consumed samples:     31692800 | consumed tokens:  64906854400 | elapsed time per iteration (s): 0.37 | learning rate: 5.469E-05 | global batch size:   256 | lm loss: 3.244065E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.625 | TFLOPs: 32.24 |
7:  iteration   123900/  173500 | consumed samples:     31718400 | consumed tokens:  64959283200 | elapsed time per iteration (s): 0.37 | learning rate: 5.456E-05 | global batch size:   256 | lm loss: 3.238605E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.751 | TFLOPs: 32.10 |
0: [2023-03-17 12:10:46,160] [INFO] [logging.py:68:log_dist] [Rank 0] step=124000, skipped=0, lr=[5.443416434803536e-05, 5.443416434803536e-05, 5.443416434803536e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   124000/  173500 | consumed samples:     31744000 | consumed tokens:  65011712000 | elapsed time per iteration (s): 0.37 | learning rate: 5.443E-05 | global batch size:   256 | lm loss: 3.241483E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.038 | TFLOPs: 32.16 |
0: steps: 124000 loss: 3.2384 iter time (s): 0.370 samples/sec: 692.098
7:  iteration   124100/  173500 | consumed samples:     31769600 | consumed tokens:  65064140800 | elapsed time per iteration (s): 0.37 | learning rate: 5.430E-05 | global batch size:   256 | lm loss: 3.240224E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.723 | TFLOPs: 32.33 |
7:  iteration   124200/  173500 | consumed samples:     31795200 | consumed tokens:  65116569600 | elapsed time per iteration (s): 0.37 | learning rate: 5.418E-05 | global batch size:   256 | lm loss: 3.239789E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.550 | TFLOPs: 32.37 |
7:  iteration   124300/  173500 | consumed samples:     31820800 | consumed tokens:  65168998400 | elapsed time per iteration (s): 0.37 | learning rate: 5.405E-05 | global batch size:   256 | lm loss: 3.238081E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.108 | TFLOPs: 32.31 |
7:  iteration   124400/  173500 | consumed samples:     31846400 | consumed tokens:  65221427200 | elapsed time per iteration (s): 0.37 | learning rate: 5.392E-05 | global batch size:   256 | lm loss: 3.240784E+00 | grad norm: 0.247 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.311 | TFLOPs: 32.36 |
7:  iteration   124500/  173500 | consumed samples:     31872000 | consumed tokens:  65273856000 | elapsed time per iteration (s): 0.37 | learning rate: 5.379E-05 | global batch size:   256 | lm loss: 3.244986E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.620 | TFLOPs: 32.10 |
7:  iteration   124600/  173500 | consumed samples:     31897600 | consumed tokens:  65326284800 | elapsed time per iteration (s): 0.37 | learning rate: 5.366E-05 | global batch size:   256 | lm loss: 3.241081E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.207 | TFLOPs: 32.36 |
7:  iteration   124700/  173500 | consumed samples:     31923200 | consumed tokens:  65378713600 | elapsed time per iteration (s): 0.37 | learning rate: 5.353E-05 | global batch size:   256 | lm loss: 3.242035E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.357 | TFLOPs: 32.41 |
7:  iteration   124800/  173500 | consumed samples:     31948800 | consumed tokens:  65431142400 | elapsed time per iteration (s): 0.37 | learning rate: 5.340E-05 | global batch size:   256 | lm loss: 3.238276E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.023 | TFLOPs: 32.44 |
7:  iteration   124900/  173500 | consumed samples:     31974400 | consumed tokens:  65483571200 | elapsed time per iteration (s): 0.37 | learning rate: 5.328E-05 | global batch size:   256 | lm loss: 3.240077E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.702 | TFLOPs: 32.43 |
7:  iteration   125000/  173500 | consumed samples:     32000000 | consumed tokens:  65536000000 | elapsed time per iteration (s): 0.37 | learning rate: 5.315E-05 | global batch size:   256 | lm loss: 3.241554E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.900 | TFLOPs: 32.44 |
7:  iteration   125100/  173500 | consumed samples:     32025600 | consumed tokens:  65588428800 | elapsed time per iteration (s): 0.37 | learning rate: 5.302E-05 | global batch size:   256 | lm loss: 3.236672E+00 | grad norm: 0.210 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.987 | TFLOPs: 32.44 |
7:  iteration   125200/  173500 | consumed samples:     32051200 | consumed tokens:  65640857600 | elapsed time per iteration (s): 0.37 | learning rate: 5.289E-05 | global batch size:   256 | lm loss: 3.242884E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.592 | TFLOPs: 32.42 |
7:  iteration   125300/  173500 | consumed samples:     32076800 | consumed tokens:  65693286400 | elapsed time per iteration (s): 0.37 | learning rate: 5.277E-05 | global batch size:   256 | lm loss: 3.239657E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.377 | TFLOPs: 32.41 |
7:  iteration   125400/  173500 | consumed samples:     32102400 | consumed tokens:  65745715200 | elapsed time per iteration (s): 0.37 | learning rate: 5.264E-05 | global batch size:   256 | lm loss: 3.236136E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.649 | TFLOPs: 32.47 |
7:  iteration   125500/  173500 | consumed samples:     32128000 | consumed tokens:  65798144000 | elapsed time per iteration (s): 0.37 | learning rate: 5.251E-05 | global batch size:   256 | lm loss: 3.239732E+00 | grad norm: 0.235 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.341 | TFLOPs: 32.46 |
7:  iteration   125600/  173500 | consumed samples:     32153600 | consumed tokens:  65850572800 | elapsed time per iteration (s): 0.37 | learning rate: 5.239E-05 | global batch size:   256 | lm loss: 3.236299E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.127 | TFLOPs: 32.40 |
7:  iteration   125700/  173500 | consumed samples:     32179200 | consumed tokens:  65903001600 | elapsed time per iteration (s): 0.37 | learning rate: 5.226E-05 | global batch size:   256 | lm loss: 3.238719E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.026 | TFLOPs: 32.44 |
7:  iteration   125800/  173500 | consumed samples:     32204800 | consumed tokens:  65955430400 | elapsed time per iteration (s): 0.37 | learning rate: 5.213E-05 | global batch size:   256 | lm loss: 3.236402E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.978 | TFLOPs: 32.44 |
7:  iteration   125900/  173500 | consumed samples:     32230400 | consumed tokens:  66007859200 | elapsed time per iteration (s): 0.37 | learning rate: 5.201E-05 | global batch size:   256 | lm loss: 3.238153E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.798 | TFLOPs: 32.43 |
0: [2023-03-17 12:23:03,909] [INFO] [logging.py:68:log_dist] [Rank 0] step=126000, skipped=0, lr=[5.188210163686188e-05, 5.188210163686188e-05, 5.188210163686188e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   126000/  173500 | consumed samples:     32256000 | consumed tokens:  66060288000 | elapsed time per iteration (s): 0.37 | learning rate: 5.188E-05 | global batch size:   256 | lm loss: 3.237169E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.803 | TFLOPs: 32.43 |
0: steps: 126000 loss: 3.2410 iter time (s): 0.367 samples/sec: 697.852
7:  iteration   126100/  173500 | consumed samples:     32281600 | consumed tokens:  66112716800 | elapsed time per iteration (s): 0.37 | learning rate: 5.176E-05 | global batch size:   256 | lm loss: 3.240836E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 687.307 | TFLOPs: 32.08 |
7:  iteration   126200/  173500 | consumed samples:     32307200 | consumed tokens:  66165145600 | elapsed time per iteration (s): 0.37 | learning rate: 5.163E-05 | global batch size:   256 | lm loss: 3.235494E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.722 | TFLOPs: 32.24 |
7:  iteration   126300/  173500 | consumed samples:     32332800 | consumed tokens:  66217574400 | elapsed time per iteration (s): 0.37 | learning rate: 5.151E-05 | global batch size:   256 | lm loss: 3.236462E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.064 | TFLOPs: 32.16 |
7:  iteration   126400/  173500 | consumed samples:     32358400 | consumed tokens:  66270003200 | elapsed time per iteration (s): 0.37 | learning rate: 5.138E-05 | global batch size:   256 | lm loss: 3.233070E+00 | grad norm: 0.228 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 690.102 | TFLOPs: 32.21 |
7:  iteration   126500/  173500 | consumed samples:     32384000 | consumed tokens:  66322432000 | elapsed time per iteration (s): 0.37 | learning rate: 5.126E-05 | global batch size:   256 | lm loss: 3.235772E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.304 | TFLOPs: 32.36 |
7:  iteration   126600/  173500 | consumed samples:     32409600 | consumed tokens:  66374860800 | elapsed time per iteration (s): 0.37 | learning rate: 5.113E-05 | global batch size:   256 | lm loss: 3.242638E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.117 | TFLOPs: 32.45 |
7:  iteration   126700/  173500 | consumed samples:     32435200 | consumed tokens:  66427289600 | elapsed time per iteration (s): 0.37 | learning rate: 5.101E-05 | global batch size:   256 | lm loss: 3.241402E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.041 | TFLOPs: 32.44 |
7:  iteration   126800/  173500 | consumed samples:     32460800 | consumed tokens:  66479718400 | elapsed time per iteration (s): 0.37 | learning rate: 5.088E-05 | global batch size:   256 | lm loss: 3.239292E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.137 | TFLOPs: 32.45 |
7:  iteration   126900/  173500 | consumed samples:     32486400 | consumed tokens:  66532147200 | elapsed time per iteration (s): 0.37 | learning rate: 5.076E-05 | global batch size:   256 | lm loss: 3.232467E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.898 | TFLOPs: 32.39 |
7:  iteration   127000/  173500 | consumed samples:     32512000 | consumed tokens:  66584576000 | elapsed time per iteration (s): 0.37 | learning rate: 5.064E-05 | global batch size:   256 | lm loss: 3.235983E+00 | grad norm: 0.215 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.965 | TFLOPs: 32.44 |
7:  iteration   127100/  173500 | consumed samples:     32537600 | consumed tokens:  66637004800 | elapsed time per iteration (s): 0.38 | learning rate: 5.051E-05 | global batch size:   256 | lm loss: 3.238970E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.051 | TFLOPs: 31.37 |
7:  iteration   127200/  173500 | consumed samples:     32563200 | consumed tokens:  66689433600 | elapsed time per iteration (s): 0.37 | learning rate: 5.039E-05 | global batch size:   256 | lm loss: 3.238431E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.625 | TFLOPs: 32.42 |
7:  iteration   127300/  173500 | consumed samples:     32588800 | consumed tokens:  66741862400 | elapsed time per iteration (s): 0.37 | learning rate: 5.026E-05 | global batch size:   256 | lm loss: 3.237859E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.632 | TFLOPs: 32.42 |
7:  iteration   127400/  173500 | consumed samples:     32614400 | consumed tokens:  66794291200 | elapsed time per iteration (s): 0.37 | learning rate: 5.014E-05 | global batch size:   256 | lm loss: 3.236248E+00 | grad norm: 0.209 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.601 | TFLOPs: 32.42 |
7:  iteration   127500/  173500 | consumed samples:     32640000 | consumed tokens:  66846720000 | elapsed time per iteration (s): 0.37 | learning rate: 5.002E-05 | global batch size:   256 | lm loss: 3.236500E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.836 | TFLOPs: 32.43 |
7:  iteration   127600/  173500 | consumed samples:     32665600 | consumed tokens:  66899148800 | elapsed time per iteration (s): 0.37 | learning rate: 4.990E-05 | global batch size:   256 | lm loss: 3.237938E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.662 | TFLOPs: 32.42 |
7:  iteration   127700/  173500 | consumed samples:     32691200 | consumed tokens:  66951577600 | elapsed time per iteration (s): 0.37 | learning rate: 4.977E-05 | global batch size:   256 | lm loss: 3.234972E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.065 | TFLOPs: 32.40 |
7:  iteration   127800/  173500 | consumed samples:     32716800 | consumed tokens:  67004006400 | elapsed time per iteration (s): 0.37 | learning rate: 4.965E-05 | global batch size:   256 | lm loss: 3.234755E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.883 | TFLOPs: 32.34 |
7:  iteration   127900/  173500 | consumed samples:     32742400 | consumed tokens:  67056435200 | elapsed time per iteration (s): 0.37 | learning rate: 4.953E-05 | global batch size:   256 | lm loss: 3.241638E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.487 | TFLOPs: 32.42 |
0: [2023-03-17 12:35:23,483] [INFO] [logging.py:68:log_dist] [Rank 0] step=128000, skipped=0, lr=[4.94077976375529e-05, 4.94077976375529e-05, 4.94077976375529e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   128000/  173500 | consumed samples:     32768000 | consumed tokens:  67108864000 | elapsed time per iteration (s): 0.37 | learning rate: 4.941E-05 | global batch size:   256 | lm loss: 3.236887E+00 | grad norm: 0.220 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.087 | TFLOPs: 32.44 |
0: steps: 128000 loss: 3.2640 iter time (s): 0.368 samples/sec: 696.139
7:  iteration   128100/  173500 | consumed samples:     32793600 | consumed tokens:  67161292800 | elapsed time per iteration (s): 0.37 | learning rate: 4.929E-05 | global batch size:   256 | lm loss: 3.237285E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.867 | TFLOPs: 32.43 |
7:  iteration   128200/  173500 | consumed samples:     32819200 | consumed tokens:  67213721600 | elapsed time per iteration (s): 0.37 | learning rate: 4.916E-05 | global batch size:   256 | lm loss: 3.236263E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.774 | TFLOPs: 32.15 |
7:  iteration   128300/  173500 | consumed samples:     32844800 | consumed tokens:  67266150400 | elapsed time per iteration (s): 0.37 | learning rate: 4.904E-05 | global batch size:   256 | lm loss: 3.239043E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.719 | TFLOPs: 32.43 |
7:  iteration   128400/  173500 | consumed samples:     32870400 | consumed tokens:  67318579200 | elapsed time per iteration (s): 0.37 | learning rate: 4.892E-05 | global batch size:   256 | lm loss: 3.235141E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.117 | TFLOPs: 32.45 |
7:  iteration   128500/  173500 | consumed samples:     32896000 | consumed tokens:  67371008000 | elapsed time per iteration (s): 0.37 | learning rate: 4.880E-05 | global batch size:   256 | lm loss: 3.239670E+00 | grad norm: 0.225 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.708 | TFLOPs: 32.38 |
7:  iteration   128600/  173500 | consumed samples:     32921600 | consumed tokens:  67423436800 | elapsed time per iteration (s): 0.37 | learning rate: 4.868E-05 | global batch size:   256 | lm loss: 3.236996E+00 | grad norm: 0.212 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.005 | TFLOPs: 32.44 |
7:  iteration   128700/  173500 | consumed samples:     32947200 | consumed tokens:  67475865600 | elapsed time per iteration (s): 0.37 | learning rate: 4.856E-05 | global batch size:   256 | lm loss: 3.232693E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.400 | TFLOPs: 32.41 |
7:  iteration   128800/  173500 | consumed samples:     32972800 | consumed tokens:  67528294400 | elapsed time per iteration (s): 0.37 | learning rate: 4.844E-05 | global batch size:   256 | lm loss: 3.238697E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.122 | TFLOPs: 32.31 |
7:  iteration   128900/  173500 | consumed samples:     32998400 | consumed tokens:  67580723200 | elapsed time per iteration (s): 0.37 | learning rate: 4.832E-05 | global batch size:   256 | lm loss: 3.238153E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.060 | TFLOPs: 32.44 |
7:  iteration   129000/  173500 | consumed samples:     33024000 | consumed tokens:  67633152000 | elapsed time per iteration (s): 0.37 | learning rate: 4.820E-05 | global batch size:   256 | lm loss: 3.238010E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.982 | TFLOPs: 32.44 |
7:  iteration   129100/  173500 | consumed samples:     33049600 | consumed tokens:  67685580800 | elapsed time per iteration (s): 0.37 | learning rate: 4.808E-05 | global batch size:   256 | lm loss: 3.234204E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.226 | TFLOPs: 32.45 |
7:  iteration   129200/  173500 | consumed samples:     33075200 | consumed tokens:  67738009600 | elapsed time per iteration (s): 0.37 | learning rate: 4.796E-05 | global batch size:   256 | lm loss: 3.235887E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.693 | TFLOPs: 32.47 |
7:  iteration   129300/  173500 | consumed samples:     33100800 | consumed tokens:  67790438400 | elapsed time per iteration (s): 0.37 | learning rate: 4.784E-05 | global batch size:   256 | lm loss: 3.233311E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.232 | TFLOPs: 32.45 |
7:  iteration   129400/  173500 | consumed samples:     33126400 | consumed tokens:  67842867200 | elapsed time per iteration (s): 0.37 | learning rate: 4.772E-05 | global batch size:   256 | lm loss: 3.239158E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.191 | TFLOPs: 32.45 |
7:  iteration   129500/  173500 | consumed samples:     33152000 | consumed tokens:  67895296000 | elapsed time per iteration (s): 0.37 | learning rate: 4.761E-05 | global batch size:   256 | lm loss: 3.238047E+00 | grad norm: 0.243 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.589 | TFLOPs: 32.42 |
7:  iteration   129600/  173500 | consumed samples:     33177600 | consumed tokens:  67947724800 | elapsed time per iteration (s): 0.37 | learning rate: 4.749E-05 | global batch size:   256 | lm loss: 3.235079E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.919 | TFLOPs: 32.44 |
7:  iteration   129700/  173500 | consumed samples:     33203200 | consumed tokens:  68000153600 | elapsed time per iteration (s): 0.37 | learning rate: 4.737E-05 | global batch size:   256 | lm loss: 3.238343E+00 | grad norm: 0.213 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.318 | TFLOPs: 32.13 |
7:  iteration   129800/  173500 | consumed samples:     33228800 | consumed tokens:  68052582400 | elapsed time per iteration (s): 0.37 | learning rate: 4.725E-05 | global batch size:   256 | lm loss: 3.234004E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.116 | TFLOPs: 32.45 |
7:  iteration   129900/  173500 | consumed samples:     33254400 | consumed tokens:  68105011200 | elapsed time per iteration (s): 0.37 | learning rate: 4.713E-05 | global batch size:   256 | lm loss: 3.233541E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.938 | TFLOPs: 32.44 |
0: [2023-03-17 12:47:41,074] [INFO] [logging.py:68:log_dist] [Rank 0] step=130000, skipped=0, lr=[4.7014562839599005e-05, 4.7014562839599005e-05, 4.7014562839599005e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   130000/  173500 | consumed samples:     33280000 | consumed tokens:  68157440000 | elapsed time per iteration (s): 0.37 | learning rate: 4.701E-05 | global batch size:   256 | lm loss: 3.237840E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.179 | TFLOPs: 32.45 |
0: steps: 130000 loss: 3.2390 iter time (s): 0.367 samples/sec: 698.053
7: -------------------------------------------------------------------------------------------------
7: validation loss at iteration 130000 | lm loss value: 3.379757E+00 | lm loss PPL: 2.936365E+01 | 
7: -------------------------------------------------------------------------------------------------
0: saving checkpoint at iteration  130000 to checkpoints_146m91b400m
0: [2023-03-17 12:47:41,200] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step130000 is begin to save!
0: [2023-03-17 12:47:41,204] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_01-model_00-model_states.pt...
0: [2023-03-17 12:47:41,291] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_01-model_00-model_states.pt.
0: [2023-03-17 12:47:41,291] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_03-model_00-model_states.pt...
0: [2023-03-17 12:47:41,307] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_03-model_00-model_states.pt.
0: [2023-03-17 12:47:41,308] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_04-model_00-model_states.pt...
0: [2023-03-17 12:47:41,323] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_04-model_00-model_states.pt.
0: [2023-03-17 12:47:41,323] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_05-model_00-model_states.pt...
0: [2023-03-17 12:47:41,338] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_05-model_00-model_states.pt.
0: [2023-03-17 12:47:41,338] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_06-model_00-model_states.pt...
0: [2023-03-17 12:47:41,353] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_06-model_00-model_states.pt.
0: [2023-03-17 12:47:41,353] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_07-model_00-model_states.pt...
0: [2023-03-17 12:47:41,368] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_07-model_00-model_states.pt.
0: [2023-03-17 12:47:41,368] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_08-model_00-model_states.pt...
0: [2023-03-17 12:47:41,383] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_08-model_00-model_states.pt.
0: [2023-03-17 12:47:41,383] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_09-model_00-model_states.pt...
0: [2023-03-17 12:47:41,398] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_09-model_00-model_states.pt.
0: [2023-03-17 12:47:41,398] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_10-model_00-model_states.pt...
0: [2023-03-17 12:47:41,413] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_10-model_00-model_states.pt.
0: [2023-03-17 12:47:41,413] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_11-model_00-model_states.pt...
0: [2023-03-17 12:47:41,427] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_11-model_00-model_states.pt.
0: [2023-03-17 12:47:41,428] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_12-model_00-model_states.pt...
0: [2023-03-17 12:47:41,442] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_12-model_00-model_states.pt.
0: [2023-03-17 12:47:41,443] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_13-model_00-model_states.pt...
0: [2023-03-17 12:47:41,457] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_13-model_00-model_states.pt.
0: [2023-03-17 12:47:41,458] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_14-model_00-model_states.pt...
0: [2023-03-17 12:47:41,472] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_14-model_00-model_states.pt.
0: [2023-03-17 12:47:41,473] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_15-model_00-model_states.pt...
0: [2023-03-17 12:47:41,487] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_15-model_00-model_states.pt.
0: [2023-03-17 12:47:41,488] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_16-model_00-model_states.pt...
0: [2023-03-17 12:47:41,502] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_16-model_00-model_states.pt.
0: [2023-03-17 12:47:41,503] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_17-model_00-model_states.pt...
0: [2023-03-17 12:47:41,517] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_17-model_00-model_states.pt.
0: [2023-03-17 12:47:41,518] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/layer_19-model_00-model_states.pt...
0: [2023-03-17 12:47:41,519] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/layer_19-model_00-model_states.pt.
0: [2023-03-17 12:47:41,519] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_146m91b400m/global_step130000/mp_rank_00_model_states.pt
0: [2023-03-17 12:47:41,519] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/mp_rank_00_model_states.pt...
0: [2023-03-17 12:47:41,521] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/mp_rank_00_model_states.pt.
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt...
7: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt...
1: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt...
2: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt...
5: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt...
4: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt...
3: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt...
6: [2023-03-17 12:47:41,539] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt...
0: [2023-03-17 12:47:41,575] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,576] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,576] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,576] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,576] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,577] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,577] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,577] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,577] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,577] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,578] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,578] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,578] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,579] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,579] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,579] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,580] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,580] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,580] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,580] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,580] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,580] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,581] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,581] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,581] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,581] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,581] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,581] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,584] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt.
0: [2023-03-17 12:47:41,584] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,584] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,587] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,588] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,588] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,588] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,588] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt.
6: [2023-03-17 12:47:41,588] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,588] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt
6: [2023-03-17 12:47:41,588] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
6: [2023-03-17 12:47:41,588] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0: [2023-03-17 12:47:41,604] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt
0: [2023-03-17 12:47:41,604] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt.
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
1: [2023-03-17 12:47:41,607] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt.
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
7: [2023-03-17 12:47:41,609] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,644] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,644] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,644] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,644] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,645] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,645] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,645] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,645] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,645] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,646] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,646] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,646] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,647] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,647] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,647] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,647] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,647] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,647] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,648] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,648] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,648] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,648] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,650] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,650] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,650] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,651] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,651] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,651] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,652] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,652] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,652] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,652] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,653] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,653] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,653] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,653] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,654] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,654] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
2: [2023-03-17 12:47:41,654] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt.
2: [2023-03-17 12:47:41,655] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt
2: [2023-03-17 12:47:41,655] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,656] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt.
5: [2023-03-17 12:47:41,656] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt
5: [2023-03-17 12:47:41,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
5: [2023-03-17 12:47:41,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt.
3: [2023-03-17 12:47:41,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
3: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,660] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,660] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,661] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,661] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
4: [2023-03-17 12:47:41,661] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt.
4: [2023-03-17 12:47:41,662] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_146m91b400m/global_step130000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt
4: [2023-03-17 12:47:41,662] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step130000 is ready now!
0:   successfully saved checkpoint at iteration  130000 to checkpoints_146m91b400m
7: time (ms) | save-checkpoint: 467.43
7:  iteration   130100/  173500 | consumed samples:     33305600 | consumed tokens:  68209868800 | elapsed time per iteration (s): 0.37 | learning rate: 4.690E-05 | global batch size:   256 | lm loss: 3.234792E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 683.083 | TFLOPs: 31.88 |
7:  iteration   130200/  173500 | consumed samples:     33331200 | consumed tokens:  68262297600 | elapsed time per iteration (s): 0.37 | learning rate: 4.678E-05 | global batch size:   256 | lm loss: 3.237316E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.549 | TFLOPs: 32.42 |
7:  iteration   130300/  173500 | consumed samples:     33356800 | consumed tokens:  68314726400 | elapsed time per iteration (s): 0.37 | learning rate: 4.666E-05 | global batch size:   256 | lm loss: 3.238694E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.802 | TFLOPs: 32.38 |
7:  iteration   130400/  173500 | consumed samples:     33382400 | consumed tokens:  68367155200 | elapsed time per iteration (s): 0.37 | learning rate: 4.655E-05 | global batch size:   256 | lm loss: 3.235756E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.219 | TFLOPs: 32.17 |
7:  iteration   130500/  173500 | consumed samples:     33408000 | consumed tokens:  68419584000 | elapsed time per iteration (s): 0.37 | learning rate: 4.643E-05 | global batch size:   256 | lm loss: 3.230005E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.731 | TFLOPs: 32.47 |
7:  iteration   130600/  173500 | consumed samples:     33433600 | consumed tokens:  68472012800 | elapsed time per iteration (s): 0.37 | learning rate: 4.631E-05 | global batch size:   256 | lm loss: 3.236064E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.124 | TFLOPs: 32.40 |
7:  iteration   130700/  173500 | consumed samples:     33459200 | consumed tokens:  68524441600 | elapsed time per iteration (s): 0.37 | learning rate: 4.620E-05 | global batch size:   256 | lm loss: 3.235036E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.920 | TFLOPs: 31.88 |
7:  iteration   130800/  173500 | consumed samples:     33484800 | consumed tokens:  68576870400 | elapsed time per iteration (s): 0.37 | learning rate: 4.608E-05 | global batch size:   256 | lm loss: 3.232506E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 695.464 | TFLOPs: 32.46 |
7:  iteration   130900/  173500 | consumed samples:     33510400 | consumed tokens:  68629299200 | elapsed time per iteration (s): 0.37 | learning rate: 4.596E-05 | global batch size:   256 | lm loss: 3.232732E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.603 | TFLOPs: 32.37 |
7:  iteration   131000/  173500 | consumed samples:     33536000 | consumed tokens:  68681728000 | elapsed time per iteration (s): 0.37 | learning rate: 4.585E-05 | global batch size:   256 | lm loss: 3.234077E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.069 | TFLOPs: 32.40 |
7:  iteration   131100/  173500 | consumed samples:     33561600 | consumed tokens:  68734156800 | elapsed time per iteration (s): 0.39 | learning rate: 4.573E-05 | global batch size:   256 | lm loss: 3.235398E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 649.192 | TFLOPs: 30.30 |
7:  iteration   131200/  173500 | consumed samples:     33587200 | consumed tokens:  68786585600 | elapsed time per iteration (s): 0.38 | learning rate: 4.562E-05 | global batch size:   256 | lm loss: 3.235623E+00 | grad norm: 0.226 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 670.580 | TFLOPs: 31.30 |
7:  iteration   131300/  173500 | consumed samples:     33612800 | consumed tokens:  68839014400 | elapsed time per iteration (s): 0.38 | learning rate: 4.550E-05 | global batch size:   256 | lm loss: 3.227113E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 679.548 | TFLOPs: 31.72 |
7:  iteration   131400/  173500 | consumed samples:     33638400 | consumed tokens:  68891443200 | elapsed time per iteration (s): 0.38 | learning rate: 4.539E-05 | global batch size:   256 | lm loss: 3.234945E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 672.754 | TFLOPs: 31.40 |
7:  iteration   131500/  173500 | consumed samples:     33664000 | consumed tokens:  68943872000 | elapsed time per iteration (s): 0.38 | learning rate: 4.527E-05 | global batch size:   256 | lm loss: 3.230286E+00 | grad norm: 0.230 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 671.523 | TFLOPs: 31.34 |
7:  iteration   131600/  173500 | consumed samples:     33689600 | consumed tokens:  68996300800 | elapsed time per iteration (s): 0.38 | learning rate: 4.516E-05 | global batch size:   256 | lm loss: 3.231218E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 667.590 | TFLOPs: 31.16 |
7:  iteration   131700/  173500 | consumed samples:     33715200 | consumed tokens:  69048729600 | elapsed time per iteration (s): 0.38 | learning rate: 4.505E-05 | global batch size:   256 | lm loss: 3.240941E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 680.158 | TFLOPs: 31.75 |
7:  iteration   131800/  173500 | consumed samples:     33740800 | consumed tokens:  69101158400 | elapsed time per iteration (s): 0.38 | learning rate: 4.493E-05 | global batch size:   256 | lm loss: 3.232294E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 681.811 | TFLOPs: 31.82 |
7:  iteration   131900/  173500 | consumed samples:     33766400 | consumed tokens:  69153587200 | elapsed time per iteration (s): 0.38 | learning rate: 4.482E-05 | global batch size:   256 | lm loss: 3.234278E+00 | grad norm: 0.223 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 678.542 | TFLOPs: 31.67 |
0: [2023-03-17 13:00:11,438] [INFO] [logging.py:68:log_dist] [Rank 0] step=132000, skipped=0, lr=[4.4705599266134565e-05, 4.4705599266134565e-05, 4.4705599266134565e-05], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)]
7:  iteration   132000/  173500 | consumed samples:     33792000 | consumed tokens:  69206016000 | elapsed time per iteration (s): 0.38 | learning rate: 4.471E-05 | global batch size:   256 | lm loss: 3.232729E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 682.611 | TFLOPs: 31.86 |
0: steps: 132000 loss: 3.2903 iter time (s): 0.373 samples/sec: 685.594
7:  iteration   132100/  173500 | consumed samples:     33817600 | consumed tokens:  69258444800 | elapsed time per iteration (s): 0.37 | learning rate: 4.459E-05 | global batch size:   256 | lm loss: 3.234491E+00 | grad norm: 0.232 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 688.176 | TFLOPs: 32.12 |
7:  iteration   132200/  173500 | consumed samples:     33843200 | consumed tokens:  69310873600 | elapsed time per iteration (s): 0.37 | learning rate: 4.448E-05 | global batch size:   256 | lm loss: 3.231437E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.191 | TFLOPs: 32.31 |
7:  iteration   132300/  173500 | consumed samples:     33868800 | consumed tokens:  69363302400 | elapsed time per iteration (s): 0.37 | learning rate: 4.437E-05 | global batch size:   256 | lm loss: 3.228429E+00 | grad norm: 0.234 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.836 | TFLOPs: 32.43 |
7:  iteration   132400/  173500 | consumed samples:     33894400 | consumed tokens:  69415731200 | elapsed time per iteration (s): 0.37 | learning rate: 4.425E-05 | global batch size:   256 | lm loss: 3.234377E+00 | grad norm: 0.218 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.489 | TFLOPs: 32.42 |
7:  iteration   132500/  173500 | consumed samples:     33920000 | consumed tokens:  69468160000 | elapsed time per iteration (s): 0.37 | learning rate: 4.414E-05 | global batch size:   256 | lm loss: 3.232939E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 692.996 | TFLOPs: 32.35 |
7:  iteration   132600/  173500 | consumed samples:     33945600 | consumed tokens:  69520588800 | elapsed time per iteration (s): 0.37 | learning rate: 4.403E-05 | global batch size:   256 | lm loss: 3.236042E+00 | grad norm: 0.224 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.181 | TFLOPs: 32.40 |
7:  iteration   132700/  173500 | consumed samples:     33971200 | consumed tokens:  69573017600 | elapsed time per iteration (s): 0.37 | learning rate: 4.392E-05 | global batch size:   256 | lm loss: 3.233426E+00 | grad norm: 0.221 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.466 | TFLOPs: 32.42 |
7:  iteration   132800/  173500 | consumed samples:     33996800 | consumed tokens:  69625446400 | elapsed time per iteration (s): 0.37 | learning rate: 4.381E-05 | global batch size:   256 | lm loss: 3.237799E+00 | grad norm: 0.217 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.683 | TFLOPs: 32.38 |
7:  iteration   132900/  173500 | consumed samples:     34022400 | consumed tokens:  69677875200 | elapsed time per iteration (s): 0.37 | learning rate: 4.369E-05 | global batch size:   256 | lm loss: 3.234265E+00 | grad norm: 0.227 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 689.714 | TFLOPs: 32.19 |
7:  iteration   133000/  173500 | consumed samples:     34048000 | consumed tokens:  69730304000 | elapsed time per iteration (s): 0.37 | learning rate: 4.358E-05 | global batch size:   256 | lm loss: 3.229533E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.539 | TFLOPs: 32.37 |
7:  iteration   133100/  173500 | consumed samples:     34073600 | consumed tokens:  69782732800 | elapsed time per iteration (s): 0.37 | learning rate: 4.347E-05 | global batch size:   256 | lm loss: 3.233245E+00 | grad norm: 0.222 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.482 | TFLOPs: 32.42 |
7:  iteration   133200/  173500 | consumed samples:     34099200 | consumed tokens:  69835161600 | elapsed time per iteration (s): 0.37 | learning rate: 4.336E-05 | global batch size:   256 | lm loss: 3.238348E+00 | grad norm: 0.219 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.783 | TFLOPs: 32.43 |
7:  iteration   133300/  173500 | consumed samples:     34124800 | consumed tokens:  69887590400 | elapsed time per iteration (s): 0.37 | learning rate: 4.325E-05 | global batch size:   256 | lm loss: 3.230676E+00 | grad norm: 0.216 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.705 | TFLOPs: 32.43 |
7:  iteration   133400/  173500 | consumed samples:     34150400 | consumed tokens:  69940019200 | elapsed time per iteration (s): 0.37 | learning rate: 4.314E-05 | global batch size:   256 | lm loss: 3.230516E+00 | grad norm: 0.229 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.898 | TFLOPs: 32.44 |
7:  iteration   133500/  173500 | consumed samples:     34176000 | consumed tokens:  69992448000 | elapsed time per iteration (s): 0.37 | learning rate: 4.303E-05 | global batch size:   256 | lm loss: 3.236269E+00 | grad norm: 0.231 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 694.716 | TFLOPs: 32.43 |
7:  iteration   133600/  173500 | consumed samples:     34201600 | consumed tokens:  70044876800 | elapsed time per iteration (s): 0.37 | learning rate: 4.292E-05 | global batch size:   256 | lm loss: 3.234643E+00 | grad norm: 0.214 | num zeros: 0.0 | number of skipped iterations:   0 | number of nan iterations:   0 | samples per second: 693.610 | TFLOPs: 32.38 |