10/15/2023 00:08:47 - INFO - app - Distributed environment: NO Num processes: 1 Process index: 0 Local process index: 0 Device: cuda Mixed precision type: no 10/15/2023 00:08:52 - INFO - app - Num examples = 5 10/15/2023 00:08:52 - INFO - app - Num batches each epoch = 5 10/15/2023 00:08:52 - INFO - app - Num Epochs = 40 10/15/2023 00:08:52 - INFO - app - Instantaneous batch size per device = 1 10/15/2023 00:08:52 - INFO - app - Total train batch size (w. parallel, distributed & accumulation) = 1 10/15/2023 00:08:52 - INFO - app - Gradient Accumulation steps = 1 10/15/2023 00:08:52 - INFO - app - Training Steps = 200 10/15/2023 00:08:52 - INFO - app - STEP = 1 10/15/2023 00:09:00 - INFO - app - STEP = 2 10/15/2023 00:09:07 - INFO - app - STEP = 3 10/15/2023 00:09:13 - INFO - app - STEP = 4 10/15/2023 00:09:20 - INFO - app - STEP = 5 10/15/2023 00:09:26 - INFO - app - Running validation... Generating 4 images with prompt: A photo of sks dog in a bucket. 10/15/2023 00:09:52 - INFO - app - STEP = 6 10/15/2023 00:09:58 - INFO - app - STEP = 7 10/15/2023 00:10:05 - INFO - app - STEP = 8 10/15/2023 00:10:12 - INFO - app - STEP = 9 10/15/2023 00:10:18 - INFO - app - STEP = 10 10/15/2023 00:10:25 - INFO - app - STEP = 11 10/15/2023 00:10:32 - INFO - app - STEP = 12 10/15/2023 00:10:39 - INFO - app - STEP = 13 10/15/2023 00:10:45 - INFO - app - STEP = 14 10/15/2023 00:10:52 - INFO - app - STEP = 15 10/15/2023 00:10:59 - INFO - app - STEP = 16 10/15/2023 00:11:06 - INFO - app - STEP = 17 10/15/2023 00:11:12 - INFO - app - STEP = 18 10/15/2023 00:11:19 - INFO - app - STEP = 19 10/15/2023 00:11:26 - INFO - app - STEP = 20 10/15/2023 00:11:32 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-20 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-20/optimizer.bin 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-20/scheduler.bin 10/15/2023 00:11:32 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-20/random_states_0.pkl 10/15/2023 00:11:32 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-20 10/15/2023 00:11:33 - INFO - app - STEP = 21 10/15/2023 00:11:39 - INFO - app - STEP = 22 10/15/2023 00:11:46 - INFO - app - STEP = 23 10/15/2023 00:11:53 - INFO - app - STEP = 24 10/15/2023 00:11:59 - INFO - app - STEP = 25 10/15/2023 00:12:06 - INFO - app - STEP = 26 10/15/2023 00:12:13 - INFO - app - STEP = 27 10/15/2023 00:12:20 - INFO - app - STEP = 28 10/15/2023 00:12:27 - INFO - app - STEP = 29 10/15/2023 00:12:33 - INFO - app - STEP = 30 10/15/2023 00:12:40 - INFO - app - STEP = 31 10/15/2023 00:12:47 - INFO - app - STEP = 32 10/15/2023 00:12:53 - INFO - app - STEP = 33 10/15/2023 00:13:00 - INFO - app - STEP = 34 10/15/2023 00:13:07 - INFO - app - STEP = 35 10/15/2023 00:13:14 - INFO - app - STEP = 36 10/15/2023 00:13:20 - INFO - app - STEP = 37 10/15/2023 00:13:27 - INFO - app - STEP = 38 10/15/2023 00:13:34 - INFO - app - STEP = 39 10/15/2023 00:13:40 - INFO - app - STEP = 40 10/15/2023 00:13:47 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-40 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-40/optimizer.bin 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-40/scheduler.bin 10/15/2023 00:13:47 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-40/random_states_0.pkl 10/15/2023 00:13:47 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-40 10/15/2023 00:13:47 - INFO - app - STEP = 41 10/15/2023 00:13:54 - INFO - app - STEP = 42 10/15/2023 00:14:01 - INFO - app - STEP = 43 10/15/2023 00:14:08 - INFO - app - STEP = 44 10/15/2023 00:14:14 - INFO - app - STEP = 45 10/15/2023 00:14:21 - INFO - app - STEP = 46 10/15/2023 00:14:28 - INFO - app - STEP = 47 10/15/2023 00:14:35 - INFO - app - STEP = 48 10/15/2023 00:14:41 - INFO - app - STEP = 49 10/15/2023 00:14:48 - INFO - app - STEP = 50 10/15/2023 00:14:55 - INFO - app - STEP = 51 10/15/2023 00:15:01 - INFO - app - STEP = 52 10/15/2023 00:15:08 - INFO - app - STEP = 53 10/15/2023 00:15:15 - INFO - app - STEP = 54 10/15/2023 00:15:22 - INFO - app - STEP = 55 10/15/2023 00:15:28 - INFO - app - STEP = 56 10/15/2023 00:15:35 - INFO - app - STEP = 57 10/15/2023 00:15:42 - INFO - app - STEP = 58 10/15/2023 00:15:49 - INFO - app - STEP = 59 10/15/2023 00:15:55 - INFO - app - STEP = 60 10/15/2023 00:16:02 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-60 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-60/optimizer.bin 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-60/scheduler.bin 10/15/2023 00:16:02 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-60/random_states_0.pkl 10/15/2023 00:16:02 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-60 10/15/2023 00:16:02 - INFO - app - STEP = 61 10/15/2023 00:16:09 - INFO - app - STEP = 62 10/15/2023 00:16:16 - INFO - app - STEP = 63 10/15/2023 00:16:22 - INFO - app - STEP = 64 10/15/2023 00:16:29 - INFO - app - STEP = 65 10/15/2023 00:16:36 - INFO - app - STEP = 66 10/15/2023 00:16:42 - INFO - app - STEP = 67 10/15/2023 00:16:49 - INFO - app - STEP = 68 10/15/2023 00:16:56 - INFO - app - STEP = 69 10/15/2023 00:17:03 - INFO - app - STEP = 70 10/15/2023 00:17:09 - INFO - app - STEP = 71 10/15/2023 00:17:16 - INFO - app - STEP = 72 10/15/2023 00:17:23 - INFO - app - STEP = 73 10/15/2023 00:17:30 - INFO - app - STEP = 74 10/15/2023 00:17:36 - INFO - app - STEP = 75 10/15/2023 00:17:43 - INFO - app - STEP = 76 10/15/2023 00:17:50 - INFO - app - STEP = 77 10/15/2023 00:17:56 - INFO - app - STEP = 78 10/15/2023 00:18:03 - INFO - app - STEP = 79 10/15/2023 00:18:10 - INFO - app - STEP = 80 10/15/2023 00:18:16 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-80 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-80/optimizer.bin 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-80/scheduler.bin 10/15/2023 00:18:16 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-80/random_states_0.pkl 10/15/2023 00:18:16 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-80 10/15/2023 00:18:17 - INFO - app - STEP = 81 10/15/2023 00:18:23 - INFO - app - STEP = 82 10/15/2023 00:18:30 - INFO - app - STEP = 83 10/15/2023 00:18:37 - INFO - app - STEP = 84 10/15/2023 00:18:44 - INFO - app - STEP = 85 10/15/2023 00:18:50 - INFO - app - STEP = 86 10/15/2023 00:18:57 - INFO - app - STEP = 87 10/15/2023 00:19:04 - INFO - app - STEP = 88 10/15/2023 00:19:11 - INFO - app - STEP = 89 10/15/2023 00:19:17 - INFO - app - STEP = 90 10/15/2023 00:19:24 - INFO - app - STEP = 91 10/15/2023 00:19:31 - INFO - app - STEP = 92 10/15/2023 00:19:37 - INFO - app - STEP = 93 10/15/2023 00:19:44 - INFO - app - STEP = 94 10/15/2023 00:19:51 - INFO - app - STEP = 95 10/15/2023 00:19:58 - INFO - app - STEP = 96 10/15/2023 00:20:04 - INFO - app - STEP = 97 10/15/2023 00:20:11 - INFO - app - STEP = 98 10/15/2023 00:20:18 - INFO - app - STEP = 99 10/15/2023 00:20:24 - INFO - app - STEP = 100 10/15/2023 00:20:31 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-100 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-100/optimizer.bin 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-100/scheduler.bin 10/15/2023 00:20:31 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-100/random_states_0.pkl 10/15/2023 00:20:31 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-100 10/15/2023 00:20:31 - INFO - app - STEP = 101 10/15/2023 00:20:38 - INFO - app - STEP = 102 10/15/2023 00:20:45 - INFO - app - STEP = 103 10/15/2023 00:20:51 - INFO - app - STEP = 104 10/15/2023 00:20:58 - INFO - app - STEP = 105 10/15/2023 00:21:05 - INFO - app - STEP = 106 10/15/2023 00:21:12 - INFO - app - STEP = 107 10/15/2023 00:21:18 - INFO - app - STEP = 108 10/15/2023 00:21:25 - INFO - app - STEP = 109 10/15/2023 00:21:32 - INFO - app - STEP = 110 10/15/2023 00:21:39 - INFO - app - STEP = 111 10/15/2023 00:21:45 - INFO - app - STEP = 112 10/15/2023 00:21:52 - INFO - app - STEP = 113 10/15/2023 00:21:59 - INFO - app - STEP = 114 10/15/2023 00:22:05 - INFO - app - STEP = 115 10/15/2023 00:22:12 - INFO - app - STEP = 116 10/15/2023 00:22:19 - INFO - app - STEP = 117 10/15/2023 00:22:26 - INFO - app - STEP = 118 10/15/2023 00:22:32 - INFO - app - STEP = 119 10/15/2023 00:22:39 - INFO - app - STEP = 120 10/15/2023 00:22:46 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-120 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-120/optimizer.bin 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-120/scheduler.bin 10/15/2023 00:22:46 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-120/random_states_0.pkl 10/15/2023 00:22:46 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-120 10/15/2023 00:22:46 - INFO - app - STEP = 121 10/15/2023 00:22:53 - INFO - app - STEP = 122 10/15/2023 00:22:59 - INFO - app - STEP = 123 10/15/2023 00:23:06 - INFO - app - STEP = 124 10/15/2023 00:23:13 - INFO - app - STEP = 125 10/15/2023 00:23:20 - INFO - app - STEP = 126 10/15/2023 00:23:26 - INFO - app - STEP = 127 10/15/2023 00:23:33 - INFO - app - STEP = 128 10/15/2023 00:23:40 - INFO - app - STEP = 129 10/15/2023 00:23:46 - INFO - app - STEP = 130 10/15/2023 00:23:53 - INFO - app - STEP = 131 10/15/2023 00:24:00 - INFO - app - STEP = 132 10/15/2023 00:24:07 - INFO - app - STEP = 133 10/15/2023 00:24:13 - INFO - app - STEP = 134 10/15/2023 00:24:20 - INFO - app - STEP = 135 10/15/2023 00:24:27 - INFO - app - STEP = 136 10/15/2023 00:24:34 - INFO - app - STEP = 137 10/15/2023 00:24:40 - INFO - app - STEP = 138 10/15/2023 00:24:47 - INFO - app - STEP = 139 10/15/2023 00:24:54 - INFO - app - STEP = 140 10/15/2023 00:25:00 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-140 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-140/optimizer.bin 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-140/scheduler.bin 10/15/2023 00:25:00 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-140/random_states_0.pkl 10/15/2023 00:25:00 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-140 10/15/2023 00:25:01 - INFO - app - STEP = 141 10/15/2023 00:25:07 - INFO - app - STEP = 142 10/15/2023 00:25:14 - INFO - app - STEP = 143 10/15/2023 00:25:21 - INFO - app - STEP = 144 10/15/2023 00:25:27 - INFO - app - STEP = 145 10/15/2023 00:25:34 - INFO - app - STEP = 146 10/15/2023 00:25:41 - INFO - app - STEP = 147 10/15/2023 00:25:48 - INFO - app - STEP = 148 10/15/2023 00:25:54 - INFO - app - STEP = 149 10/15/2023 00:26:01 - INFO - app - STEP = 150 10/15/2023 00:26:08 - INFO - app - STEP = 151 10/15/2023 00:26:15 - INFO - app - STEP = 152 10/15/2023 00:26:21 - INFO - app - STEP = 153 10/15/2023 00:26:28 - INFO - app - STEP = 154 10/15/2023 00:26:35 - INFO - app - STEP = 155 10/15/2023 00:26:42 - INFO - app - STEP = 156 10/15/2023 00:26:48 - INFO - app - STEP = 157 10/15/2023 00:26:55 - INFO - app - STEP = 158 10/15/2023 00:27:02 - INFO - app - STEP = 159 10/15/2023 00:27:08 - INFO - app - STEP = 160 10/15/2023 00:27:15 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-160 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-160/optimizer.bin 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-160/scheduler.bin 10/15/2023 00:27:15 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-160/random_states_0.pkl 10/15/2023 00:27:15 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-160 10/15/2023 00:27:15 - INFO - app - STEP = 161 10/15/2023 00:27:22 - INFO - app - STEP = 162 10/15/2023 00:27:29 - INFO - app - STEP = 163 10/15/2023 00:27:36 - INFO - app - STEP = 164 10/15/2023 00:27:42 - INFO - app - STEP = 165 10/15/2023 00:27:49 - INFO - app - STEP = 166 10/15/2023 00:27:56 - INFO - app - STEP = 167 10/15/2023 00:28:03 - INFO - app - STEP = 168 10/15/2023 00:28:09 - INFO - app - STEP = 169 10/15/2023 00:28:16 - INFO - app - STEP = 170 10/15/2023 00:28:23 - INFO - app - STEP = 171 10/15/2023 00:28:29 - INFO - app - STEP = 172 10/15/2023 00:28:36 - INFO - app - STEP = 173 10/15/2023 00:28:43 - INFO - app - STEP = 174 10/15/2023 00:28:50 - INFO - app - STEP = 175 10/15/2023 00:28:56 - INFO - app - STEP = 176 10/15/2023 00:29:03 - INFO - app - STEP = 177 10/15/2023 00:29:10 - INFO - app - STEP = 178 10/15/2023 00:29:17 - INFO - app - STEP = 179 10/15/2023 00:29:23 - INFO - app - STEP = 180 10/15/2023 00:29:30 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-180 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-180/optimizer.bin 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-180/scheduler.bin 10/15/2023 00:29:30 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-180/random_states_0.pkl 10/15/2023 00:29:30 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-180 10/15/2023 00:29:30 - INFO - app - STEP = 181 10/15/2023 00:29:37 - INFO - app - STEP = 182 10/15/2023 00:29:44 - INFO - app - STEP = 183 10/15/2023 00:29:50 - INFO - app - STEP = 184 10/15/2023 00:29:57 - INFO - app - STEP = 185 10/15/2023 00:30:04 - INFO - app - STEP = 186 10/15/2023 00:30:10 - INFO - app - STEP = 187 10/15/2023 00:30:17 - INFO - app - STEP = 188 10/15/2023 00:30:24 - INFO - app - STEP = 189 10/15/2023 00:30:31 - INFO - app - STEP = 190 10/15/2023 00:30:37 - INFO - app - STEP = 191 10/15/2023 00:30:44 - INFO - app - STEP = 192 10/15/2023 00:30:51 - INFO - app - STEP = 193 10/15/2023 00:30:58 - INFO - app - STEP = 194 10/15/2023 00:31:04 - INFO - app - STEP = 195 10/15/2023 00:31:11 - INFO - app - STEP = 196 10/15/2023 00:31:18 - INFO - app - STEP = 197 10/15/2023 00:31:25 - INFO - app - STEP = 198 10/15/2023 00:31:31 - INFO - app - STEP = 199 10/15/2023 00:31:38 - INFO - app - STEP = 200 10/15/2023 00:31:44 - INFO - accelerate.accelerator - Saving current state to models/SD-Dog-Test5/checkpoint-200 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Optimizer state saved in models/SD-Dog-Test5/checkpoint-200/optimizer.bin 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Scheduler state saved in models/SD-Dog-Test5/checkpoint-200/scheduler.bin 10/15/2023 00:31:45 - INFO - accelerate.checkpointing - Random states saved in models/SD-Dog-Test5/checkpoint-200/random_states_0.pkl 10/15/2023 00:31:45 - INFO - app - Saved state to models/SD-Dog-Test5/checkpoint-200