k4d3
/

yiff_toolkit

Diffusers

TensorBoard

English

Not-For-All-Audiences

Model card Files Files and versions Metrics Training metrics Community

k4d3 commited on May 29

Commit

eb8cf2c

•

1 Parent(s): ca498bf

notice of move

Browse files

Files changed (1) hide show

README.md +1 -951

README.md CHANGED Viewed

@@ -14,957 +14,7 @@ tags:
 ---
----
-## Auto Taggers
-### [eva02-vit-large-448-8046](https://huggingface.co/Thouph/eva02-vit-large-448-8046)
-You want to install the only dependency, besides torch, I mean..
-```bash
-pip install timm
-```
-The following inference script for the tagger needs a folder as input, be warned that it also converts WebP images to PNG and you can specify tags to be ignored and some other stuff! I recommend reading through it and changing whatever you need.
-[Colab Notebook](https://colab.research.google.com/drive/1gIB2fGjLAuh6s_hrNlIPCkw_3jodoFP0?usp=sharing)
-<div style="background-color: lightyellow; padding: 10px;">
-<details>
-  <summary>Click to reveal inference script</summary>
-```python
-import os
-import torch
-from torchvision import transforms
-from PIL import Image
-import json
-import re
-# Set the threshold for tag selection
-THRESHOLD = 0.3
-# Define the directory containing the images and the path to the model
-image_dir = r"./images"
-model_path = r"./model.pth"
-# Define the set of ignored tags
-ignored_tags = {"grandfathered content"}
-# Check if CUDA is available, else use CPU
-device = torch.device("cuda" if torch.cuda.is_available() else "cpu")
-# Load the model and set it to evaluation mode
-model = torch.load(model_path, map_location=device)
-model = model.to(device)
-model.eval()
-# Define the image transformations
-transform = transforms.Compose(
-    [
-        # Resize the images to 448x448
-        transforms.Resize((448, 448)),
-        # Convert the images to PyTorch tensors
-        transforms.ToTensor(),
-        # Normalize the images with the given mean and standard deviation
-        transforms.Normalize(
-            mean=[0.48145466, 0.4578275, 0.40821073],
-            std=[0.26862954, 0.26130258, 0.27577711],
-        ),
-    ]
-)
-# Load the tags from the JSON file
-with open("tags_8041.json", "r", encoding="utf8") as file:
-    tags = json.load(file)
-allowed_tags = sorted(tags)
-# Add placeholders and explicitness tags to the list of allowed tags
-allowed_tags.insert(0, "placeholder0")
-allowed_tags.append("placeholder1")
-allowed_tags.append("explicit")
-allowed_tags.append("questionable")
-allowed_tags.append("safe")
-# Define the allowed image extensions
-image_exts = [".jpg", ".jpeg", ".png"]
-for filename in os.listdir(image_dir):
-    # Check if the file is a WebP image
-    if filename.endswith(".webp"):
-        # Construct the input and output file paths
-        input_path = os.path.join(image_dir, filename)
-        output_path = os.path.join(image_dir, os.path.splitext(filename)[0] + ".png")
-        # Open the WebP image and save it as a PNG
-        image = Image.open(input_path)
-        image.save(output_path, "PNG")
-        print(f"Converted {filename} to {os.path.basename(output_path)}")
-        # Delete the original WebP image
-        os.remove(input_path)
-        print(f"Deleted {filename}")
-# Get the list of image files in the directory
-image_files = [
-    file
-    for file in os.listdir(image_dir)
-    if os.path.splitext(file)[1].lower() in image_exts
-]
-for image_filename in image_files:
-    image_path = os.path.join(image_dir, image_filename)
-    # Open the image
-    img = Image.open(image_path)
-    # If the image has an alpha channel, replace it with black
-    if img.mode in ("RGBA", "LA") or (img.mode == "P" and "transparency" in img.info):
-        alpha = Image.new(
-            "L", img.size, 0
-        )  # Create alpha image with mode 'L' (8-bit grayscale)
-        alpha = alpha.convert(img.mode)  # Convert alpha image to same mode as img
-        img = Image.alpha_composite(alpha, img)
-    # Convert the image to RGB
-    img = img.convert("RGB")
-    # Apply the transformations and move the tensor to the device
-    tensor = transform(img).unsqueeze(0).to(device)
-    # Make a forward pass through the model and get the output
-    with torch.no_grad():
-        out = model(tensor)
-    # Apply the sigmoid function to the output to get probabilities
-    probabilities = torch.sigmoid(out[0])
-    # Get the indices of the tags with probabilities above the threshold
-    indices = torch.where(probabilities > THRESHOLD)[0]
-    values = probabilities[indices]
-    # Sort the indices by the corresponding probabilities in descending order
-    sorted_indices = torch.argsort(values, descending=True)
-    # Get the tags corresponding to the sorted indices, excluding ignored tags and replacing underscores with spaces
-    tags_to_write = [
-        allowed_tags[indices[i]].replace("_", " ")
-        for i in sorted_indices
-        if allowed_tags[indices[i]] not in ignored_tags
-        and allowed_tags[indices[i]] not in ("placeholder0", "placeholder1")
-    ]
-    # Replace 'safe', 'explicit', and 'questionable' with their 'rating_' counterparts
-    tags_to_write = [
-        tag.replace("safe", "rating_safe")
-        .replace("explicit", "rating_explicit")
-        .replace("questionable", "rating_questionable")
-        for tag in tags_to_write
-    ]
-    # Escape unescaped parentheses in the tags
-    tags_to_write_escaped = [
-        re.sub(r"(?<!\\)(\(|\))", r"\\\1", tag) for tag in tags_to_write
-    ]
-    # Create a text file for each image with the filtered and escaped tags
-    text_filename = os.path.splitext(image_filename)[0] + ".txt"
-    text_path = os.path.join(image_dir, text_filename)
-    with open(text_path, "w", encoding="utf8") as text_file:
-        text_file.write(", ".join(tags_to_write_escaped))
-```
-</details>
-</div>
-## LoRA Training Guide
-### Installation Tips
----
-Firstly, download kohya_ss' [sd-scripts](https://github.com/kohya-ss/sd-scripts), you need to set up your environment either like [this](https://github.com/kohya-ss/sd-scripts?tab=readme-ov-file#windows-installation) tells you for Windows, or if you are using Linux or Miniconda on Windows, you are probably smart enough to figure out the installation for it. I recommend always installing the latest [PyTorch](https://pytorch.org/get-started/locally/) in the virtual environment you are going to use, which at the time of writing is `2.2.2`. I hope future me has faster PyTorch!
-Ok, just in case you aren't smart enough how to install the sd-scripts under Miniconda for Windows I actually "guided" someone recently, just so I can tell you about it:
-```bash
-# Installing sd-scripts
-git clone https://github.com/kohya-ss/sd-scripts
-cd sd-scripts
-# Creating the conda environment and installing requirements
-conda create -n sdscripts python=3.10.14
-conda activate sdscripts
-conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
-python -m pip install --use-pep517 --upgrade -r requirements.txt
-python -m pip install --use-pep517 lycoris_lora
-accelerate config
-```
-`accelerate config` will ask you a bunch of questions, you need to actually read each one and reply with the truth. In most cases the truth looks like this: `This machine, No distributed training, no, no, no, all, fp16`.
-You might also want to install `xformers` or `bitsandbytes`.
-```bash
-# Installing xformers
-# Use the same command just replace 'xformers' with any other package you may need.
-python -m pip install --use-pep517 xformers
-# Installing bitsandbytes for windows
-python -m pip install --use-pep517 bitsandbytes --index-url=https://jllllll.github.io/bitsandbytes-windows-webui
-```
----
-### Pony Training
----
-I'm not going to lie, it is a bit complicated to explain everything. But here is my best attempt going through some "basic" stuff and almost all lines in order.
-#### Download Pony in Diffusers Format
-I'm using the diffusers version for training I converted, you can download it using `git`.
-```bash
-git clone https://huggingface.co/k4d3/ponydiffusers
-```
----
-#### Sample Prompt File
-A sample prompt file is used during training to sample images. A sample prompt for example might look like this for Pony:
-```py
-# anthro female kindred
-score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, female anthro kindred, mask, presenting, white pillow, bedroom, looking at viewer, detailed background, amazing_background, scenery porn, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40
-# anthro female wolf
-score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry, solo, anthro female wolf, sexy pose, standing, gray fur, brown fur, canine pussy, black nose, blue eyes, pink areola, pink nipples, detailed background, amazing_background, realistic, photo --n low quality, worst quality, blurred background, blurry, simple background --w 1024 --h 1024 --d 1 --l 6.0 --s 40
-```
-Please note that sample prompts should not exceed 77 tokens, you can use [Count Tokens in Sample Prompts](https://huggingface.co/k4d3/yiff_toolkit/blob/main/dataset_tools/Count%20Tokens%20in%20Sample%20Prompts.ipynb) from [/dataset_tools](https://huggingface.co/k4d3/yiff_toolkit/tree/main/dataset_tools) to analyze your prompts.
-If you are training with multiple GPUs, ensure that the total number of prompts is divisible by the number of GPUs without any remainder or a card will idle.
----
-#### Training Commands
----
-##### `accelerate launch`
-For two GPUs:
-```python
-accelerate launch --num_processes=2 --multi_gpu --num_machines=1 --gpu_ids=0,1 --num_cpu_threads_per_process=2  "./sdxl_train_network.py"
-```
-Single GPU:
-```python
-accelerate launch --num_cpu_threads_per_process=2 "./sdxl_train_network.py"
-```
----
-&nbsp;
-And now lets break down a bunch of arguments we can pass to `sd-scripts`.
-&nbsp;
-##### `--lowram`
-If you are running running out of system memory like I do with 2 GPUs and a really fat model that gets loaded into it per GPU, this option will help you save a bit of it and might get you out of OOM hell.
----
-##### `--pretrained_model_name_or_path`
-The directory containing the checkpoint you just downloaded. I recommend closing the path if you are using a local diffusers model with a `/`. You can also specify a `.safetensors` or `.ckpt` if that is what you have!
-```python
-    --pretrained_model_name_or_path="/ponydiffusers/"
-```
----
-##### `--output_dir`
-This is where all the saved epochs or steps will be saved, including the last one. If y
-```python
-    --output_dir="/output_dir"
-```
----
-##### `--train_data_dir`
-The directory containing the dataset. We prepared this earlier together.
-```python
-    --train_data_dir="/training_dir"
-```
----
-##### `--resolution`
-Always set this to match the model's resolution, which in Pony's case it is 1024x1024. If you can't fit into the VRAM, you can decrease it to `512,512` as a last resort.
-```python
-    --resolution="1024,1024"
-```
----
-##### `--enable_bucket`
-Creates different buckets by pre-categorizing images with different aspect ratios into different buckets. This technique helps to avoid issues like unnatural crops that are common when models are trained to produce square images. This allows the creation of batches where every item has the same size, but the image size of batches may differ.
----
-##### `--min_bucket_reso` and `--max_bucket_reso`
-Specifies the minimum and maximum resolutions used by the buckets. These values are ignored if `--bucket_no_upscale` is set.
-```python
-    --min_bucket_reso=256 --max_bucket_reso=1024
-```
----
-##### `--network_alpha`
-Specifies how many of the trained Network Ranks are allowed to alter the base model.
-```python
-    --network_alpha=4
-```
----
-##### `--save_model_as`
-You can use this to specify either `ckpt` or `safetensors` for the file format.
-```python
-    --save_model_as="safetensors"
-```
----
-##### `--network_module`
-Specifies which network module you are going to train.
-```python
-    --network_module="lycoris.kohya"
-```
----
-##### `--network_args`
-The arguments passed down to the network.
-```python
-    --network_args \
-               "use_reentrant=False" \
-               "preset=full" \
-               "conv_dim=256" \
-               "conv_alpha=4" \
-               "use_tucker=False" \
-               "use_scalar=False" \
-               "rank_dropout_scale=False" \
-               "algo=locon" \
-               "train_norm=False" \
-               "block_dims=8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8" \
-               "block_alphas=0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625" \
-```
-**Let's break it down!**
----
-###### `preset`
-The [Preset](https://github.com/KohakuBlueleaf/LyCORIS/blob/HEAD/docs/Preset.md)/config system added to LyCORIS for more fine-grained control.
-- `full`
-  - default preset, train all the layers in the UNet and CLIP.
-- `full-lin`
-  - `full` but skip convolutional layers.
-- `attn-mlp`
-  - "kohya preset", train all the transformer block.
-- `attn-only`
-  - only attention layer will be trained, lot of papers only do training on attn layer.
-- `unet-transformer-only`
-  - as same as kohya_ss/sd_scripts with disabled TE, or, attn-mlp preset with train_unet_only enabled.
-- `unet-convblock-only`
-  - only ResBlock, UpSample, DownSample will be trained.
----
-###### `conv_dim` and `conv_alpha`
-The convolution dimensions are related to the rank of the convolution in the model, adjusting this value can have a [significant impact](https://ashejunius.com/alpha-and-dimensions-two-wild-settings-of-training-lora-in-stable-diffusion-d7ad3e3a3b0a) and lowering it affected the aesthetic differences between different LoRA samples. and an alpha value of `128` was used for training a specific character's face while Kohaku recommended to set this to `1` for both LoCon and LoHa.
-```python
-conv_block_dims = [conv_dim] * num_total_blocks
-conv_block_alphas = [conv_alpha] * num_total_blocks
-```
----
-###### `module_dropout` and `dropout` and `rank_dropout`
-[![An AI generated image.](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)](https://huggingface.co/k4d3/yiff_toolkit/resolve/main/static/tutorial/dropout1.png)
-`rank_dropout` is a form of dropout, which is a regularization technique used in neural networks to prevent overfitting and improve generalization. However, unlike traditional dropout which randomly sets a proportion of inputs to zero, `rank_dropout` operates on the rank of the input tensor `lx`. First a binary mask is created with the same rank as `lx` with each element set to `True` with probability `1 - rank_dropout` and `False` otherwise. Then the `mask` is applied to `lx` to randomly set some of its elements to zero. After applying the dropout, a scaling factor is applied to `lx` to compensate for the dropped out elements. This is done to ensure that the expected sum of `lx` remains the same before and after dropout. The scaling factor is `1.0 / (1.0 - self.rank_dropout)`.
-It’s called “rank” dropout because it operates on the rank of the input tensor, rather than its individual elements. This can be particularly useful in tasks where the rank of the input is important.
-If `rank_dropout` is set to `0`, it means that no dropout is applied to the rank of the input tensor `lx`. All elements of the mask would be set to `True` and when the mask gets applied to `lx` all of it's elements would be retained and when the scaling factor is applied after dropout it's value would just equal `self.scale` because `1.0 / (1.0 - 0)` is `1`. Basically, setting this to `0` effectively disables the dropout mechanism but it will still do some meaningless calculations, and you can't set it to None, so if you really want to disable dropouts simply don't specify them! 😇
-```python
-def forward(self, x):
-    org_forwarded = self.org_forward(x)
-    # module dropout
-    if self.module_dropout is not None and self.training:
-        if torch.rand(1) < self.module_dropout:
-            return org_forwarded
-    lx = self.lora_down(x)
-    # normal dropout
-    if self.dropout is not None and self.training:
-        lx = torch.nn.functional.dropout(lx, p=self.dropout)
-    # rank dropout
-    if self.rank_dropout is not None and self.training:
-        mask = torch.rand((lx.size(0), self.lora_dim), device=lx.device) > self.rank_dropout
-        if len(lx.size()) == 3:
-            mask = mask.unsqueeze(1)
-        elif len(lx.size()) == 4:
-            mask = mask.unsqueeze(-1).unsqueeze(-1)
-        lx = lx * mask
-        scale = self.scale * (1.0 / (1.0 - self.rank_dropout))
-    else:
-        scale = self.scale
-    lx = self.lora_up(lx)
-    return org_forwarded + lx * self.multiplier * scale
-```
-The network you are training needs to support it though! See [PR#545](https://github.com/kohya-ss/sd-scripts/pull/545) for more details.
----
-###### `use_tucker`
-Can be used for all but `(IA)^3` and native fine-tuning.
-Tucker decomposition is a method in mathematics that decomposes a tensor into a set of matrices and one small core tensor reducing the computational complexity and memory requirements of the model. It is used in various LyCORIS modules on various blocks. In LoCon for example, if `use_tucker` is `True` and the kernel size `k_size` is not `(1, 1)`, then the convolution operation is decomposed into three separate operations.
-1. A 1x1 convolution that reduces the number of channels from `in_dim` to `lora_dim`.
-2. A convolution with the original kernel size `k_size`, stride `stride`, and padding `padding`, but with a reduced number of channels `lora_dim`.
-3. A 1x1 convolution that increases the number of channels back from `lora_dim` to `out_dim`.
-If `use_tucker` is `False` or not set, or if the kernel size k_size is `(1, 1)`, then a standard convolution operation is performed with the original kernel size, stride, and padding, and the number of channels is reduced from `in_dim` to `lora_dim`.
----
-###### `use_scalar`
-An additional learned parameter that scales the contribution of the low-rank weights before they are added to the original weights. This scalar can control the extent to which the low-rank adaptation modifies the original weights. By training this scalar, the model can learn the optimal balance between preserving the original pre-trained weights and allowing for low-rank adaptation.
-```python
-# Check if the 'use_scalar' flag is set to True
-if use_scalar:
-    # If True, initialize a learnable parameter 'scalar' with a starting value of 0.0.
-    # This parameter will be optimized during the training process.
-    self.scalar = nn.Parameter(torch.tensor(0.0))
-else:
-    # If the 'use_scalar' flag is False, set 'scalar' to a fixed value of 1.0.
-    # This means the low-rank weights will be added to the original weights without scaling.
-    self.scalar = torch.tensor(1.0)
-```
-The `use_scalar` flag allows the model to determine how much influence the low-rank weights should have on the final weights. If `use_scalar` is `True`, the model can learn the optimal value for `self.scalar` during training, which multiplies the low-rank weights before they are added to the original weights. This provides a way to balance between the original pre-trained weights and the new low-rank adaptations, potentially leading to better performance and more efficient training. The initial value of `0.0` for `self.scalar` suggests that the model starts with no contribution from the low-rank weights and learns the appropriate scale during training.
----
-###### `rank_dropout_scale`
-A boolean flag that determines whether to scale the dropout mask to have an average value of `1` or not. This is particularly useful when you want to maintain the original scale of the tensor values after applying dropout, which can be important for the stability of the training process.
-```python
-def forward(self, orig_weight, org_bias, new_weight, new_bias, *args, **kwargs):
-    # Retrieve the device that the 'oft_blocks' tensor is on. This ensures that any new tensors created are on the same device.
-    device = self.oft_blocks.device
-    # Check if rank dropout is enabled and the model is in training mode.
-    if self.rank_dropout and self.training:
-        # Create a random tensor the same shape as 'oft_blocks', with values drawn from a uniform distribution.
-        # Then create a dropout mask by checking if each value is less than 'self.rank_dropout' probability.
-        drop = (torch.rand(self.oft_blocks, device=device) < self.rank_dropout).to(
-            self.oft_blocks.dtype
-        )
-        # If 'rank_dropout_scale' is True, scale the dropout mask to have an average value of 1.
-        # This helps maintain the scale of the tensor's values after dropout is applied.
-        if self.rank_dropout_scale:
-            drop /= drop.mean()
-    else:
-        # If rank dropout is not enabled or the model is not in training mode, set 'drop' to 1 (no dropout).
-        drop = 1
-```
----
-###### `algo`
-The LyCORIS algorithm used, you can find a [list](https://github.com/KohakuBlueleaf/LyCORIS/blob/HEAD/docs/Algo-List.md) of the implemented algorithms and an [explanation](https://github.com/KohakuBlueleaf/LyCORIS/blob/HEAD/docs/Algo-Details.md) of them, with a [demo](https://github.com/KohakuBlueleaf/LyCORIS/blob/HEAD/docs/Demo.md) you can also dig into the [research paper](https://arxiv.org/pdf/2309.14859.pdf).
----
-###### `train_norm`
-Controls whether to train normalization layers used by all algorithms except `(IA)^3` or not.
----
-###### `block_dims`
-Specify the rank of each block, it takes exactly 25 numbers, that is why this line is so long.
----
-###### `block_alphas`
-Specifies the alpha of each block, this too also takes 25 numbers if you don't specify it `network_alpha` will be used instead for the value.
----
-That concludes the `network_args`.
----
-##### `--network_dropout`
-This float controls the drop of neurons out of training every step, `0` or `None` is default behavior (no dropout), 1 would drop all neurons. Using `weight_decompose=True` will ignore `network_dropout` and only rank and module dropout will be applied.
-```python
-    --network_dropout=0 \
-```
----
-##### `--lr_scheduler`
-A learning rate scheduler in PyTorch is a tool that adjusts the learning rate during the training process. It’s used to modulate the learning rate in response to how the model is performing, which can lead to increased performance and reduced training time.
-Possible values: `linear`, `cosine`, `cosine_with_restarts`, `polynomial`, `constant` (default), `constant_with_warmup`, `adafactor`
-Note, `adafactor` scheduler can only be used with the `adafactor` optimizer!
-```python
-    --lr_scheduler="cosine" \
-```
----
-##### `--lr_scheduler_num_cycles`
-Number of restarts for cosine scheduler with restarts. It isn't used by any other scheduler.
-```py
-    --lr_scheduler_num_cycles=1 \
-```
----
-##### `--learning_rate` and `--unet_lr` and `--text_encoder_lr`
-The learning rate determines how much the weights of the network are updated in response to the estimated error each time the weights are updated. If the learning rate is too large, the weights may overshoot the optimal solution. If it’s too small, the weights may get stuck in a suboptimal solution.
-For AdamW the optimal LR seems to be `0.0001` or `1e-4` if you want to impress your friends.
-```py
-    --learning_rate=0.0001 --unet_lr=0.0001 --text_encoder_lr=0.0001
-```
----
-##### `--network_dim`
-The Network Rank (Dimension) is responsible for how many features your LoRA will be training. It is in a close relation with Network Alpha and the Unet + TE learning rates and of course the quality of your dataset. Personal experimentation with these values is strongly recommended.
-```py
-    --network_dim=8
-```
----
-##### `--output_name`
-Specify the output name excluding the file extension.
-**WARNING**: If for some reason this is ever left empty your last epoch won't be saved!
-```py
-    --output_name="last"
-```
----
-##### `--scale_weight_norms`
-Max-norm regularization is a technique that constrains the norm of the incoming weight vector at each hidden unit to be upper bounded by a fixed constant. It prevents the weights from growing too large and helps improve the performance of stochastic gradient descent training of deep neural nets.
-Dropout affects the network architecture without changing the weights, while Max-Norm Regularization directly modifies the weights of the network. Both techniques are used to prevent overfitting and improve the generalization of the model. You can learn more about both in this [research paper](https://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf).
-```py
-    --scale_weight_norms=1.0
-```
----
-##### `--max_grad_norm`
-Also known as Gradient Clipping, if you notice that gradients are exploding during training (loss becomes NaN or very large), consider adjusting the `--max_grad_norm` parameter, it operates on the gradients during the backpropagation process, while `--scale_weight_norms` operates on the weights of the neural network. This allows them to complement each other and provide a more robust approach to stabilizing the learning process and improving model performance.
-```py
-    --max_grad_norm=1.0
-```
----
-##### `--no_half_vae`
-Disables mixed precision for the SDXL VAE and sets it to `float32`. Very useful if you don't like NaNs.
----
-##### `--save_every_n_epochs` and `--save_last_n_epochs` or `--save_every_n_steps` and `--save_last_n_steps`
-- `--save_every_n_steps` and `--save_every_n_epochs`: A LoRA file will be created at each n-th step or epoch specified here.
-- `--save_last_n_steps` and `--save_last_n_epochs`: Discards every saved file except for the last `n` you specify here.
-Learning will always end with what you specify in `--max_train_epochs` or `--max_train_steps`.
-```py
-    --save_every_n_epochs=50
-```
----
-##### `--mixed_precision`
-⚠️
-```py
-    --mixed_precision="fp16"
-```
----
-##### `--save_precision`
-⚠️
-```py
-    --save_precision="fp16"
-```
----
-##### `--caption_extension`
-The file extension for caption files. Default is `.caption`. These caption files contain text descriptions that are associated with the training images. When you run the training script, it will look for files with this specified extension in the training data folder. The script uses the content of these files as captions to provide context for the images during the training process.
-For example, if your images are named `image1.jpg`, `image2.jpg`, and so on, and you use the default .caption extension, the script will expect the caption files to be named `image1.caption`, `image2.caption`, etc. If you want to use a different extension, like `.txt`, you would set the caption_extension parameter to `.txt`, and the script would then look for `image1.txt`, `image2.txt`, and so on.
-```py
-    --caption_extension=".txt"
-```
-##### `--cache_latents` and `--cache_latents_to_disk`
-⚠️
-```py
-    --cache_latents --cache_latents_to_disk
-```
----
-##### `--optimizer_type`
-The default optimizer is `AdamW` and there are a bunch of them added every month or so, therefore I'm not listing them all, you can find the list if you really want, but `AdamW` is the best as of this writing so we use that!
-```py
-    --optimizer_type="AdamW"
-```
----
-##### `--dataset_repeats`
-Repeats the dataset when training with captions, by default it is set to `1` so we'll set this to `0` with:
-```py
-    --dataset_repeats=0
-```
----
-##### `--max_train_steps`
-Specify the number of steps or epochs to train. If both `--max_train_steps` and `--max_train_epochs` are specified, the number of epochs takes precedence.
-```py
-    --max_train_steps=400
-```
----
-##### `--shuffle_caption`
-Shuffles the captions set by `--caption_separator`, it is a comma `,` by default which will work perfectly for our case since our captions look like this:
-> rating_questionable, 5 fingers, anthro, bent over, big breasts, blue eyes, blue hair, breasts, butt, claws, curved horn, female, finger claws, fingers, fur, hair, huge breasts, looking at viewer, looking back, looking back at viewer, nipples, nude, pink body, pink hair, pink nipples, rear view, solo, tail, tail tuft, tuft, by lunarii, by x-leon-x, mythology, krystal \(darkmaster781\), dragon, scalie, wickerbeast, The image showcases a pink-scaled wickerbeast a furred dragon creature with blue eyes., She has large breasts and a thick tail., Her blue and pink horns are curved and pointy and she has a slight smiling expression on her face., Her scales are shiny and she has a blue and pink pattern on her body., Her hair is a mix of pink and blue., She is looking back at the viewer with a curious expression., She has a slight blush.,
-As you can tell, I have separated the caption part not just the tags with a `,` to make sure everything gets shuffled. I'm at this point pretty certain this is beneficial especially when your caption file contains more than 77 tokens.
-NOTE: `--cache_text_encoder_outputs` and `--cache_text_encoder_outputs_to_disk` can't be used together with `--shuffle_caption`. Both of these aim to reduce VRAM usage, you will need to decide between these yourself!
----
-##### `--sdpa` or `--xformers` or `--mem_eff_attn`
-Each of these options modifies the attention mechanism used in the model, which can have a significant impact on the model's performance and memory usage. The choice between `--xformers` or `--mem_eff_attn` and `--spda` will depend on your GPU. You can benchmark it by repeating a training with them!
-- `--xformers`: This flag enables the use of XFormers in the model. XFormers is a library developed by Facebook Research that provides a collection of transformer models optimized for different hardware and use-cases. These models are designed to be highly efficient, flexible, and customizable. They offer various types of attention mechanisms and other features that can be beneficial in scenarios where you have limited GPU memory or need to handle large-scale data.
-- `--mem_eff_attn`: This flag enables the use of memory-efficient attention mechanisms in the model. The memory-efficient attention is designed to reduce the memory footprint during the training of transformer models, which can be particularly beneficial when working with large models or datasets.
-- `--sdpa`: This option enables the use of Scaled Dot-Product Attention (SDPA) within the model. SDPA is a fundamental component of transformer models that calculates the attention scores between queries and keys. It scales the dot products by the dimensionality of the keys to stabilize gradients during training. This mechanism is particularly useful for handling long sequences and can potentially improve the model’s ability to capture long-range dependencies.
-```python
-    --sdpa
-```
----
-##### `--multires_noise_iterations` and `--multires_noise_discount`
-Multi-resolution noise is a new approach that adds noise at multiple resolutions to an image or latent image during the training of diffusion models. A model trained with this technique can generate visually striking images with a distinct aesthetic compared to the usual outputs of diffusion models.
-A model trained with multi-resolution noise can generate a more diverse range of images than regular stable diffusion, including extremely light or dark images. These have historically been challenging to achieve without resorting to using a large number of sampling steps.
-This technique is particularly beneficial when working with small datasets but you I don't think you should ever not use it.
-The `--multires_noise_discount` parameter controls the extent to which the noise amount at each resolution is weakened. A value of 0.1 is recommended. The `--multires_noise_iterations` parameter determines the number of iterations for adding multi-resolution noise, with a recommended range of 6 to 10.
-Please note that `--multires_noise_discount` has no effect without `--multires_noise_iterations`.
-###### Implementation Details
-The `get_noise_noisy_latents_and_timesteps` function samples noise that will be added to the latents. If `args.noise_offset` is true, it applies a noise offset. If `args.multires_noise_iterations` is true, it applies multi-resolution noise to the sampled noise.
-The function then samples a random timestep for each image and adds noise to the latents according to the noise magnitude at each timestep. This is the forward diffusion process.
-The `pyramid_noise_like` function generates noise with a pyramid structure. It starts with the original noise and adds upscaled noise at decreasing resolutions. The noise at each level is scaled by a discount factor raised to the power of the level. The noise is then scaled back to roughly unit variance. This function is used to implement the multi-resolution noise.
-```python
-    --multires_noise_iterations=10 --multires_noise_discount=0.1
-```
----
-##### `--sample_prompts` and `--sample_sampler` and `--sample_every_n_steps`
-You have the option of generating images during training so you can check the progress, the argument let's you pick between different samplers, by default it is on `ddim`, so you better change it!
-You can also use `--sample_every_n_epochs` instead which will take precedence over steps. The `k_` prefix means karras and the `_a` suffix means ancestral.
-```py
-    --sample_prompts=/training_dir/sample-prompts.txt --sample_sampler="euler_a" --sample_every_n_steps=100
-```
-My recommendation for Pony is to use `euler_a` for toony and for realistic `k_dpm_2`.
-Your sampler options include the following:
-```bash
-ddim, pndm, lms, euler, euler_a, heun, dpm_2, dpm_2_a, dpmsolver, dpmsolver++, dpmsingle, k_lms, k_euler, k_euler_a, k_dpm_2, k_dpm_2_a
-```
----
-So, the whole thing would look something like this:
-```python
-accelerate launch --num_cpu_threads_per_process=2  "./sdxl_train_network.py" \
-    --lowram \
-    --pretrained_model_name_or_path="/ponydiffusers/" \
-    --train_data_dir="/training_dir" \
-    --resolution="1024,1024" \
-    --output_dir="/output_dir" \
-    --enable_bucket \
-    --min_bucket_reso=256 \
-    --max_bucket_reso=1024 \
-    --network_alpha=4 \
-    --save_model_as="safetensors" \
-    --network_module="lycoris.kohya" \
-    --network_args \
-               "preset=full" \
-               "conv_dim=256" \
-               "conv_alpha=4" \
-               "use_tucker=False" \
-               "use_scalar=False" \
-               "rank_dropout_scale=False" \
-               "algo=locon" \
-               "train_norm=False" \
-               "block_dims=8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8,8" \
-               "block_alphas=0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625,0.0625" \
-    --network_dropout=0 \
-    --lr_scheduler="cosine" \
-    --learning_rate=0.0001 \
-    --unet_lr=0.0001 \
-    --text_encoder_lr=0.0001 \
-    --network_dim=8 \
-    --output_name="yifftoolkit" \
-    --scale_weight_norms=1 \
-    --no_half_vae \
-    --save_every_n_epochs=50 \
-    --mixed_precision="fp16" \
-    --save_precision="fp16" \
-    --caption_extension=".txt" \
-    --cache_latents \
-    --cache_latents_to_disk \
-    --optimizer_type="AdamW" \
-    --max_grad_norm=1 \
-    --keep_tokens=1 \
-    --max_data_loader_n_workers=8 \
-    --bucket_reso_steps=32 \
-    --multires_noise_iterations=10 \
-    --multires_noise_discount=0.1 \
-    --log_prefix=xl-locon \
-    --gradient_accumulation_steps=12 \
-    --gradient_checkpointing \
-    --train_batch_size=8 \
-    --dataset_repeats=0 \
-    --max_train_steps=400 \
-    --shuffle_caption \
-    --sdpa \
-    --sample_prompts=/training_dir/sample-prompts.txt \
-    --sample_sampler="euler_a" \
-    --sample_every_n_steps=100
-```
----
-## Embeddings for 1.5 and SDXL
-Embeddings in Stable Diffusion are high-dimensional representations of input data, such as images or text, that capture their essential features and relationships. These embeddings are used to guide the diffusion process, enabling the model to generate outputs that closely match the desired characteristics specified in the input.
-You can find in the [`/embeddings`](https://huggingface.co/k4d3/yiff_toolkit/tree/main/embeddings) folder a whole bunch of them I collected for SD 1.5 that I later converted with [this](https://huggingface.co/spaces/FoodDesert/Embedding_Converter) tool for SDXL.
-## ComfyUI Walkthrough any%
-⚠️ Coming next year! ⚠️
----
-## AnimateDiff for Masochists
-⚠️ Coming in 2026! ⚠️
----
-## Stable Cascade Furry Bible
-### Resonance Cascade
-🍆
----
-## SDXL Furry Bible
-### Some Common Knowledge Stuff
-[Resolution Lora](https://huggingface.co/jiaxiangc/res-adapter/resolve/main/sdxl-i/resolution_lora.safetensors?download=true) is a nice thing to have, it will help with consistency. For SDXL it is just a LoRA you can load in and it will do its magic. No need for a custom node or extension in this case.
-### SeaArt Furry
----
-SeaArt's furry model sadly has its cons not just pros, yes it might come with artist knowledge bundled, but it seems to have trouble doing more than one character or everyone is bad at prompting, oh and it uses raw e621 tags, which just means you have to use underscores `_` instead of spaces&nbsp;` `&nbsp; inside the tags.
-⚠️ TODO: Prompting tips.
-### Pony Diffusion V6
----
-#### Requirements
-Download the [model](https://civitai.com/models/257749/pony-diffusion-v6-xl) and load it in to whatever you use to generate images.
-#### Positive Prompt Stuff
-```python
-score_9, score_8_up, score_7_up, score_6_up, rating_explicit, source_furry,
-```
-I just assumed you wanted _explicit_ and _furry_, you can also set the rating to `rating_safe` or `rating_questionable` and the source to `source_anime`, `source_cartoon`, `source_pony`, `source_rule34` and optionally mix them however you'd like. Its your life! `score_9` is an interesting tag, the model seems to have put all it's "_artsy_" knowledge. You might want to check if it is for your taste. The other interesting tag is `score_5_up` which seems to have learned a little bit of everything regarding quality while `score_4_up` seems to be at the bottom of the autism spectrum regarding art, I do not recommend using it, but you can do whatever you want!
-You can talk to Pony in three ways, use tags only, tags are neat, but you can also just type in
-`The background is of full white marble towers in greek architecture style and a castle.` and use natural language to the fullest extent, but the best way is to mix it both, its actually recommended since the score tags by definition are tags, and you need to use them! There are also artist styles that seeped into some random tokens during training, there is a community effort by some weebs to sort them [here](https://lite.framacalc.org/4ttgzvd0rx-a6jf).
-Other nice words to have in the box depending on your mood:
-```python
-detailed background, amazing_background, scenery porn
-```
-Other types of backgrounds include:
-```python
-simple background, abstract background, spiral background, geometric background, heart background, gradient background, monotone background, pattern background, dotted background, stripped background, textured background, blurred background
-```
-After `simple background` you can also define a color for the background like `white background` to get a simple white background.
-For the character portrayal you can set many different types:
-```python
-three-quarter view, full-length portrait, headshot portrait, bust portrait, half-length portrait, torso shot
-```
-Its a good thing to describe your subject or subjects start with `solo` or `duo` or maybe `trio, group` , and then finally start describing your character in an interesting situation.
-#### Negative Prompt Stuff
-⚠️
-#### How to Prompt Female Anthro Lions
-```python
-anthro ⚠️?
-```
----
 ## Pony Diffusion V6 LoRAs

 ---
+The README has moved to [https://cringe.live/docs/yiff_toolkit/](https://cringe.live/docs/yiff_toolkit/)
 ## Pony Diffusion V6 LoRAs