README.md

---
license: mit
datasets:
- Qingyun/lmmrotate-sft-data
language:
- en
base_model:
- microsoft/Florence-2-large
pipeline_tag: image-text-to-text
tags:
- aerial
- geoscience
- remotesensing
---

<p align="center">
  <h1 align="center">LMMRotate 🎮: A Simple Aerial Detection Baseline of Multimodal Language Models</h1>
  <p align="center">
    <a href='https://scholar.google.com/citations?hl=en&user=TvsTun4AAAAJ' style='text-decoration: none' >Qingyun Li</a><sup></sup>&emsp;
    <a href='https://scholar.google.com/citations?user=A39S7JgAAAAJ&hl=en' style='text-decoration: none' >Yushi Chen</a><sup></sup>&emsp;
    <a href='https://www.researchgate.net/profile/Shu-Xinya' style='text-decoration: none' >Xinya Shu</a><sup></sup>&emsp;      
    <a href='https://scholar.google.com/citations?hl=en&user=UzPtYnQAAAAJ' style='text-decoration: none' >Dong Chen</a><sup></sup>&emsp;     
    <a href='https://scholar.google.com/citations?hl=en&user=WQgE8l8AAAAJ' style='text-decoration: none' >Xin He</a><sup></sup>&emsp;     
    <a href='https://scholar.google.com/citations?user=OYtSc4AAAAAJ&hl=en' style='text-decoration: none' >Yi Yu</a><sup></sup>&emsp;     
    <a href='https://yangxue0827.github.io/' style='text-decoration: none' >Xue Yang</a><sup></sup>&emsp;
    <p align='center'>
      If you find our work helpful, please consider giving us a ⭐!
    </p>
  </p>
</p>


- ArXiv Paper: https://arxiv.org/abs/2501.09720
- GitHub Repo: https://github.com/Li-Qingyun/mllm-mmrotate
- HuggingFace Page: https://huggingface.co/collections/Qingyun/lmmrotate-6780cabaf49c4e705023b8df

This repo hosts all the available checkpoints of Florence-2 trained for aerial detection with LMMRotate in [our paper](https://arxiv.org/abs/2501.09720).

LMMRotate is a technical practice to fine-tune Large Multimodal language Models for oriented object detection as in MMRotate and hosts the official implementation of the paper: A Simple Aerial Detection Baseline of Multimodal Language Models.

<img src="https://github.com/user-attachments/assets/d34e4c0c-9e04-446e-a511-2e7005e32074" alt="framework" width="100%" />

See the list of available checkpoint [here](https://huggingface.co/Qingyun/Florence-2-models-lmmrotate/tree/main).

The folder is named `{base_model}_vis{vision_input_size}-lang{max_language_input_length}_{dataset_name}-{annotation_version}_b{samples_per_gpu}x{num_gpus}-{num_epoch}e-{note}`

For example:

> `florence-2-b_vis1024-lang2048_dota1-train-v2_b2x16-100e-slurm-zero2`:
> - **base_model**: Microsoft/Florence-2-base
> - **vision input size**: 1024 \times 1024
> - **max language input length**: 2048
> - **aerial detection source dataset name**: dota-train (`train` split of `split_ss_dota`)
> - **annotation version**: v2 (the users should ignore this)
> - **batch size and resources**: 2x16gpus = 32
> - **schedule**: 100 epochs
> - **note**: the model is trained on a slurm cluster and accelerated with DeepSpeed ZeRO2

## Downloading Guide

You can download with your web browser on [the file page](https://huggingface.co/datasets/Qingyun/Florence-2-models-lmmrotate/tree/main).

We recommand downloading in terminal using huggingface-cli (`pip install --upgrade huggingface_cli`). You can refer to [the document](https://huggingface.co/docs/huggingface_hub/guides/download) for more usages.

```
# Set Huggingface Mirror for Chinese users (if required):
export HF_ENDPOINT=https://hf-mirror.com 
# Download a certain checkpoint:
huggingface-cli download Qingyun/Florence-2-models-lmmrotate <checkpoint_folder_name> --repo-type model --local-dir checkpoint/
# If any error (such as network error) interrupts the downloading, you just need to execute the same command, the latest huggingface_hub will resume downloading.
```

## Detection Performance

![](https://github.com/user-attachments/assets/2f45fad2-bab9-45f3-8b7f-fdd1a16db335)

## Cite

LMMRotate paper:
```
@article{li2025lmmrotate,
  title={A Simple Aerial Detection Baseline of Multimodal Language Models},
  author={Li, Qingyun and Chen, Yushi and Shu, Xinya and Chen, Dong and He, Xin and Yu Yi and Yang, Xue },
  journal={arXiv preprint arXiv:2501.09720},
  year={2025}
}
```