awoo

Signed-off-by: Balazs Horvath <[email protected]>

README.md

@@ -63,14 +63,13 @@ The Yiff Toolkit is a comprehensive set of tools designed to enhance your creati
         - [`--network_dropout`](#--network_dropout)
         - [`--lr_scheduler`](#--lr_scheduler)
         - [`--lr_scheduler_num_cycles`](#--lr_scheduler_num_cycles)
-        - [`--learning_rate`](#--learning_rate)
-        - [`--unet_lr`](#--unet_lr)
-        - [`--text_encoder_lr`](#--text_encoder_lr)
+        - [`--learning_rate` and `--unet_lr` and `--text_encoder_lr`](#--learning_rate-and---unet_lr-and---text_encoder_lr)
         - [`--network_dim`](#--network_dim)
         - [`--output_name`](#--output_name)
         - [`--scale_weight_norms`](#--scale_weight_norms)
+        - [`--max_grad_norm`](#--max_grad_norm)
         - [`--no_half_vae`](#--no_half_vae)
-        - [`--save_every_n_epochs`](#--save_every_n_epochs)
+        - [`--save_every_n_epochs` and `--save_last_n_epochs` or `--save_every_n_steps` and `--save_last_n_steps`](#--save_every_n_epochs-and---save_last_n_epochs-or---save_every_n_steps-and---save_last_n_steps)
         - [`--mixed_precision`](#--mixed_precision)
         - [`--save_precision`](#--save_precision)
         - [`--caption_extension`](#--caption_extension)
@@ -80,6 +79,7 @@ The Yiff Toolkit is a comprehensive set of tools designed to enhance your creati
         - [`--max_train_steps`](#--max_train_steps)
         - [`--shuffle_caption`](#--shuffle_caption)
         - [`--sdpa` or `--xformers` or `--mem_eff_attn`](#--sdpa-or---xformers-or---mem_eff_attn)
+        - [`--multires_noise_iterations` and `--multires_noise_discount`](#--multires_noise_iterations-and---multires_noise_discount)
         - [`--sample_prompts` and `--sample_sampler` and `--sample_every_n_steps`](#--sample_prompts-and---sample_sampler-and---sample_every_n_steps)
     - [CosXL Training](#cosxl-training)
   - [Embeddings for 1.5 and SDXL](#embeddings-for-15-and-sdxl)
@@ -485,6 +485,8 @@ If you are training with multiple GPUs, ensure that the total number of prompts
 <details>
   <summary>Click to reveal training commands.</summary>
 
+---
+
 ##### `accelerate launch`
 
 For two GPUs:
@@ -499,6 +501,8 @@ Single GPU:
 accelerate launch --num_processes=1 --num_machines=1 --gpu_ids=0 --num_cpu_threads_per_process=2 "./sdxl_train_network.py"
 ```
 
+---
+
 ##### `--lowram`
 
 If you are running running out of system memory like I do with 2 GPUs and a really fat model that gets loaded into it per GPU, this option will help you save a bit of it and might get you out of OOM hell.
@@ -660,6 +664,8 @@ conv_block_alphas = [conv_alpha] * num_total_blocks
 
 ###### `module_dropout` and `dropout` and `rank_dropout`
 
+[![An AI generated image.](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)
+
 `rank_dropout` is a form of dropout, which is a regularization technique used in neural networks to prevent overfitting and improve generalization. However, unlike traditional dropout which randomly sets a proportion of inputs to zero, `rank_dropout` operates on the rank of the input tensor `lx`. First a binary mask is created with the same rank as `lx` with each element set to `True` with probability `1 - rank_dropout` and `False` otherwise. Then the `mask` is applied to `lx` to randomly set some of its elements to zero. After applying the dropout, a scaling factor is applied to `lx` to compensate for the dropped out elements. This is done to ensure that the expected sum of `lx` remains the same before and after dropout. The scaling factor is `1.0 / (1.0 - self.rank_dropout)`.
 
 It’s called “rank” dropout because it operates on the rank of the input tensor, rather than its individual elements. This can be particularly useful in tasks where the rank of the input is important.
@@ -699,6 +705,8 @@ def forward(self, x):
     return org_forwarded + lx * self.multiplier * scale
 ```
 
+The network you are training needs to support it though! See [PR#545](https://github.com/kohya-ss/sd-scripts/pull/545) for more details.
+
 ---
 
 ###### `use_tucker`
@@ -771,9 +779,13 @@ Specifies the alpha of each block, this too also takes 25 numbers if you don't s
 
 ---
 
+That concludes the `network_args`.
+
+---
+
 ##### `--network_dropout`
 
-Using `weight_decompose=True` will ignore `network_dropout` and only rank and module dropout will be applied.
+This float controls the drop of neurons out of training every step, `0` or `None` is default behavior (no dropout), 1 would drop all neurons. Using `weight_decompose=True` will ignore `network_dropout` and only rank and module dropout will be applied.
 
 ```python
     --network_dropout=0 \
@@ -783,7 +795,11 @@ Using `weight_decompose=True` will ignore `network_dropout` and only rank and mo
 
 ##### `--lr_scheduler`
 
-⚠️
+A learning rate scheduler in PyTorch is a tool that adjusts the learning rate during the training process. It’s used to modulate the learning rate in response to how the model is performing, which can lead to increased performance and reduced training time.
+
+Possible values: `linear`, `cosine`, `cosine_with_restarts`, `polynomial`, `constant` (default), `constant_with_warmup`, `adafactor`
+
+Note, `adafactor` scheduler can only be used with the `adafactor` optimizer!
 
 ```python
     --lr_scheduler="cosine" \
@@ -793,7 +809,7 @@ Using `weight_decompose=True` will ignore `network_dropout` and only rank and mo
 
 ##### `--lr_scheduler_num_cycles`
 
-⚠️
+Number of restarts for cosine scheduler with restarts. It isn't used by any other scheduler.
 
 ```py
     --lr_scheduler_num_cycles=1 \
@@ -801,31 +817,15 @@ Using `weight_decompose=True` will ignore `network_dropout` and only rank and mo
 
 ---
 
-##### `--learning_rate`
-
-⚠️
-
-```py
-    --learning_rate=0.0001 \
-```
-
----
+##### `--learning_rate` and `--unet_lr` and `--text_encoder_lr`
 
-##### `--unet_lr`
+The learning rate determines how much the weights of the network are updated in response to the estimated error each time the weights are updated. If the learning rate is too large, the weights may overshoot the optimal solution. If it’s too small, the weights may get stuck in a suboptimal solution.
 
-⚠️
+For AdamW the optimal LR seems to be `0.0001` or `1e-4` if you want to impress your friends.
 
 ```py
+    --learning_rate=0.0001 \
     --unet_lr=0.0001 \
-```
-
----
-
-##### `--text_encoder_lr`
-
-⚠️
-
-```py
     --text_encoder_lr=0.0001 \
 ```
 
@@ -855,14 +855,22 @@ Specify the output name excluding the file extension.
 
 ##### `--scale_weight_norms`
 
-[![An AI generated image.](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)
+Max-norm regularization is a technique that constrains the norm of the incoming weight vector at each hidden unit to be upper bounded by a fixed constant. It prevents the weights from growing too large and helps improve the performance of stochastic gradient descent training of deep neural nets.
 
-Encourages the LoRA to diversify it's training by randomly removing some weights to prevent overfitting, in the real world this is called Max-norm regularization.
+Dropout affects the network architecture without changing the weights, while Max-Norm Regularization directly modifies the weights of the network. Both techniques are used to prevent overfitting and improve the generalization of the model. You can learn more about both in this [research paper](https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf).
 
-The network you are training needs to support it though! See [PR#545](https://github.com/kohya-ss/sd-scripts/pull/545) for more details.
+```py
+    --scale_weight_norms=1.0 \
+```
+
+---
+
+##### `--max_grad_norm`
+
+Also known as Gradient Clipping, if you notice that gradients are exploding during training (loss becomes NaN or very large), consider adjusting the `--max_grad_norm` parameter, it operates on the gradients during the backpropagation process, while `--scale_weight_norms` operates on the weights of the neural network. This allows them to complement each other and provide a more robust approach to stabilizing the learning process and improving model performance.
 
 ```py
-    --scale_weight_norms=1 \
+    --max_grad_norm=1.0 \
 ```
 
 ---
@@ -877,12 +885,15 @@ Disables mixed precision for the SDXL VAE and sets it to `float32`. Very useful
 
 ---
 
-##### `--save_every_n_epochs`
+##### `--save_every_n_epochs` and `--save_last_n_epochs` or `--save_every_n_steps` and `--save_last_n_steps`
 
-⚠️
+- `--save_every_n_steps` and `--save_every_n_epochs`: A LoRA file will be created at each n-th step or epoch specified here.
+- `--save_last_n_steps` and `--save_last_n_epochs`: Discards every saved file except for the last `n` you specify here.
+
+Learning will always end with what you specify in `--max_train_epochs` or `--max_train_steps`.
 
 ```py
-    --save_every_n_epochs=10 \
+    --save_every_n_epochs=50 \
 ```
 
 ---
@@ -951,7 +962,7 @@ Repeats the dataset when training with captions, by default it is set to `1` so
 Specify the number of steps or epochs to train. If both `--max_train_steps` and `--max_train_epochs` are specified, the number of epochs takes precedence.
 
 ```py
-    --max_train_steps=500 \
+    --max_train_steps=400 \
 ```
 
 ---
@@ -974,6 +985,17 @@ The choice between `--xformers` or `--mem_eff_attn` and `--spda` will depend on
 
 ---
 
+##### `--multires_noise_iterations` and `--multires_noise_discount`
+
+⚠️
+
+```python
+    --multires_noise_iterations=10 \
+    --multires_noise_discount=0.1 \
+```
+
+---
+
 ##### `--sample_prompts` and `--sample_sampler` and `--sample_every_n_steps`
 
 You have the option of generating images during training so you can check the progress, the argument let's you pick between different samplers, by default it is on `ddim`, so you better change it!
@@ -996,6 +1018,12 @@ ddim, pndm, lms, euler, euler_a, heun, dpm_2, dpm_2_a, dpmsolver, dpmsolver++, d
 
 ---
 
+So, the whole thing would look something like this:
+
+```python
+
+```
+
 </details>
 </div>