Florence-2-base-Castollux-v0.1

A microsoft/Florence-2-base finetune intended to improve the quality and formatting of image captioning.

This is a very early version as I have a limited number of training images and the variety of images is lacking. Basically all images used were high quality so captioning low quality images may perform poorly.

Uses <CAPTION> task prompt. Training on <DETAILED_CAPTION> or <MORE_DETAILED_CAPTION> did not seem to make a difference in quality compared to <CAPTION>.

Model Evaluation

Source Image	Ground Truth	Model Prediction
	A woman with dark curly hair and red lipstick is holding a red paintbrush in her right hand, with the tip of the brush touching a strand of her hair. She has dark eyeliner and is wearing a white collared shirt with a button visible. Her gaze is directed towards the viewer. Behind her is a white wall covered in various framed and unframed pieces of artwork. On the top left is a framed picture with a black frame, depicting what appears to be a still life with fruits and a glass. To the right of that is an unframed picture of white tulips in a glass vase. Further to the right is a framed picture with a black frame, showing a blue and white floral image. Below the tulips is an unframed picture of two pears on a white plate. To the right of the pears is an unframed picture of a dark blue and silver abstract form. To the right of the framed blue floral picture is an unframed picture of a white flower. To the right of the abstract form is an unframed picture of two glass bottles. A small portion of a black circular object is visible in the top left corner of the image.	A woman with dark curly hair and red lipstick is holding a red stick in her right hand, looking to the right with a slight smile. She is wearing a white collared shirt with a button-down collar. Her eyes are visible, and she has dark eyeliner and mascara. Her fingernails are painted with a light pink color. Behind her is a white wall with several framed pictures hanging on it. The first picture on the left shows a white square with a black border. The second picture in the middle shows a blue and white abstract painting. The third picture shows a black and white drawing of a white flower. The fourth picture shows an illustration of a clear glass bottle. The fifth picture has a white circular frame with a blue border. Below the first picture, there is a small piece of paper with text on it that is difficult to read.
	A gold chandelier with multiple tiers of crystals hangs in the upper center of the image. Below it, a long wooden table is set for a meal, running from the lower left towards the center right. The table is covered with a white runner. Several clear glass vases filled with pink and white flowers are placed along the center of the table. To the left of the flowers, multiple clear stemmed glasses are arranged. On either side of the flower arrangements, silver octagonal plates are placed with silver cutlery beside them. Grey cushioned chairs with wooden frames are positioned along both sides of the table. In the background, a green greenhouse structure with multiple glass panes is visible. To the left of the greenhouse, a black speaker on a stand is present. A tree with pink flowers is situated in front of the greenhouse. A glimpse of another building with a brown roof is visible in the far left background.	A long wooden table is set for a formal dinner, with a clear glass chandelier hanging from the ceiling above it. On the table, there is a vase filled with pink and white flowers, including lilies, roses, and carnations. To the left of the vase are several clear wine glasses, and to the right of the table are silverware, including spoons, forks, and knives. The table is covered with a white tablecloth. In the background, there are green gazebos and a bush with red flowers. A black speaker stand with a black speaker is visible on the left side of the image. The ground is covered in green grass. The lighting suggests it is daytime.
	Two men are in a kitchen, with the man on the left wearing a black corduroy blazer, a white button-down shirt with small black dots, and light blue jeans, holding a black coffee cup with the text "Coffee Gourmet" printed on it, and leaning against a grey countertop, smiling and looking towards the other man. The man on the right is wearing a black button-down shirt and black pants, holding a black coffee cup with the text "Coffee Gourmet" printed on it, and smiling while looking at the other man. Behind the man on the left is a stainless steel refrigerator with a water dispenser. Behind both men are grey kitchen cabinets, and a stainless steel microwave is visible on the right side of the image above the countertop. A red object is partially visible on the countertop behind the man on the left. A white door with horizontal slats is visible on the left side of the image.	Two men are standing in a kitchen, with the man on the left having short dark hair and a beard, wearing a black pinstripe blazer over a white button-down shirt and light blue jeans, and holding a black disposable coffee cup with a yellow and black design on it, and smiling at the other man, who is also smiling and looking to the right, and is wearing a dark blue long-sleeved button-up shirt and black pants, and has a black belt around his waist. Behind the men is a stainless steel refrigerator with a silver handle and a silver door. To the left of the refrigerator is a white door with vertical white shutters. Above the refrigerator are light blue kitchen cabinets with silver handles. In the background, there is a dark countertop with a red object on it. On the right side of the image, a silver microwave is visible.
	A woman with dark hair and brown eyes looks to the right, her lips slightly parted revealing her teeth. She has dark eyebrows and appears to be wearing makeup, including eyeshadow and lip gloss. A shadow falls across her face, diagonally from the top left to the bottom right, obscuring part of her forehead, nose, mouth, and chin. She is wearing a thin black strap top. The background is a gradient of light yellow to darker yellow, with a sharp, dark shadow cast on the right side of the image. The woman's left shoulder and part of her chest are visible in the lower left corner, while her right shoulder is partially visible in the lower right corner, also partially obscured by shadow.	A close-up shot shows a woman with dark, wavy hair and brown eyes, looking directly at the viewer with a neutral expression. Her hair is parted in the middle and falls over her shoulders, framing her face. Her eyes are a light brown color, and she has dark eyeliner and mascara on her eyelids. Her lips are slightly parted, revealing a light-colored lip gloss. She is wearing a dark, off-the-shoulder top that is visible at the bottom of the image. Her shadow is cast on the right side of her face, obscuring a portion of her upper body. The background is a gradient of light and dark, with a light yellow wall on the far right.
	A green gift box with a peach-colored ribbon tied in a bow and a small sprig of greenery tucked under the ribbon is held in a person's hands, with a silver bracelet visible on the wrist of the hand holding the right side of the box, and the person is wearing a light blue long-sleeved shirt. Another person, wearing a white button-down shirt, is holding a smartphone with a black case and a screen displaying what appears to be a payment interface with a dark-colored card visible on the screen. In the blurred background, there is a light green box and a brown box on a wooden surface.	A person wearing a light blue button-down shirt is holding a dark blue smartphone in their right hand and a dark green gift box in their left hand. The gift box is wrapped in dark green paper with a light brown ribbon tied in a bow at the top, and a small sprig of eucalyptus leaves attached to the top of the box. The person's fingernails are painted a light pink color. The background is blurred, but it appears to be a wooden surface with several other gift boxes stacked on top of each other.
	Three people are standing outdoors in a garden setting, where a man with short black hair and glasses wearing a navy blue suit with a red patterned tie and a white shirt is on the left, smiling and touching the hair of a man in the center. The man in the center has short black hair, a beard, and glasses, and is wearing a grey suit with a purple tie and a white patterned shirt, with his hands gesturing outwards. A woman with long black hair, wearing a pink saree with gold embellishments and carrying a gold clutch bag, is on the right, smiling and touching the ear of the man in the center. The man on the left has brown eyes, and the man in the center has brown eyes. The woman has dark eyes and is wearing gold jewelry including earrings and a necklace. The background consists of various green bushes and trees, with some blurred foliage visible. The lighting suggests it is daytime.	A man with short dark hair and glasses is on the left, wearing a dark blue and purple plaid suit with a white shirt, a maroon tie, and a silver ring on his left hand, with his right arm raised and his left arm bent at the elbow. Next to him is a man with dark hair, wearing glasses, a light grey suit jacket, a purple tie, a silver pocket square, and silver bracelets on his right wrist. A woman with long dark hair is standing on the right side of the image, facing away from the viewer and smiling. She is wearing a pink long-sleeved dress with a pink jacket over a black top, a patterned top, and gold bangles on her wrists, and is holding a beige woven purse in her right hand. The background consists of green foliage and trees, with some pink flowers visible in the upper right corner. The ground is covered in grass.
	Two women stand in a snowy area surrounded by trees, both holding bows and aiming. The woman on the left has curly dark hair and wears a light beige puffy jacket, a black top underneath, and light beige pants, along with black boots. She is looking to her left as she holds a wooden bow with a brown string, her left arm extended and her right hand pulling back the string. The woman on the right has dark hair and wears a bright red hooded sweatshirt, black leggings with white text on the side, and black boots. She is looking forward as she holds a red and black recurve bow with a green string, her left arm extended and her right hand pulling back the string. The ground is covered in snow. The background consists of tall trees with dark trunks and green foliage, some appearing to be pine trees. There is text visible on the side of the leggings of the woman on the right.	Two people are engaged in a game of archery in a snowy forest. The person on the left is wearing a white puffer jacket, white pants, and black boots, with a black belt around their waist, and is looking towards the right side of the image where the person in the red jacket is holding a bow and arrow with both hands, and the arrow has a red and black patterned design on it. The background consists of a dense forest of green pine trees, some of which are partially obscured by the snow. The ground is covered in a thick layer of snow.
	A woman with short, wavy auburn hair and fair skin is looking slightly upwards and to the left while holding a black Canon camera with a red ring around the lens. She has dark eyeliner, dark red lipstick, and is wearing a multi-strand pearl necklace and a light beige long-sleeved blouse with lace trim around the neckline. Her eyes are brown. The camera is positioned in the lower center of the image, held in both of her hands. The background is a solid green screen, with a hint of a grey ceiling or structure visible in the upper left corner. The word "Canon" is visible on the camera.	A woman with short, wavy brown hair and red lipstick is looking to the left while holding a black Canon camera in her right hand. She is wearing a cream-colored long-sleeved top with a lace trim around the neckline and a pearl necklace with multiple strands of white pearls. The camera has a black lens with a red ring around the lens. The background is a solid green color.

Training Settings

Trained with Florence-2ner using this config and ~3K images from these datasets:

• PJMixers-Images/Handpicked-Images-gemini-2.0-flash-thinking-exp-1219-CustomShareGPT
• PJMixers-Images/bghira_pseudo-camera-10k-gemini-2.0-flash-thinking-exp-1219-CustomShareGPT

{
    "model_name": "microsoft/Florence-2-base",
    "task_prompt": "<CAPTION>",
    "dataset_path": "./0000_Datasets/Gemini",
    "wandb_project_name": "Florence-2-base",
    "run_name": "Florence-2-base-Castollux-v0.1-run11",
    "epochs": 2,
    "optimizer": "CAME",
    "learning_rate": 3e-6,
    "lr_scheduler": "REX",
    "gradient_checkpointing": true,
    "freeze_vision": false,
    "freeze_language": false,
    "freeze_other": false,
    "train_batch_size": 8,
    "eval_batch_size": 8,
    "gradient_accumulation_steps": 4,
    "clip_grad_norm": 1,
    "weight_decay": 1e-2,
    "save_total_limit": 3,
    "save_steps": 10,
    "eval_steps": 10,
    "warmup_steps": 10,
    "eval_split_ratio": 0.05,
    "seed": 42,
    "filtering_processes": 128,
    "attn_implementation": "sdpa"
}