TinyOctopus: Bilingual Audio Language Model 🐙🔊

📢 Overview

TinyOctopus is a Bilingual Audio Language Model (Audio-LLM) designed to process and generate text from audio inputs. The model leverages Distil-Whisper (distil-large-v3) for audio encoding, a cross-attention projection layer for alignment, and DeepSeek 1.5B for text generation. TinyOctopus is optimized for tasks such as:

• Bilingual Automatic Speech Recognition (ASR) 🗣️
• Speech Translation 🌍
• Gender Identification 👨🧑
• Dialect Identification

TinyOctopus maintaining the architectural principles of the following structure:

🏗 Model Architecture

TinyOctopus integrates:

1. Distil-Whisper (distil-large-v3) for encoding audio inputs.
2. Cross-Attention Projection Layer (trainable) to align audio features with textual representations.
3. DeepSeek 1.5B as the core language model for text generation.

📂 Dataset

The model has been trained on multiple datasets to optimize its performance across different tasks:

• QASR Dataset: QASR is the largest transcribed Arabic speech corpus, collected from the broadcast domain. It contains 2,000 hours of multi-dialect speech sampled at 16kHz from Al Jazeera News Channel, with lightly supervised transcriptions aligned with the audio segments. Unlike previous datasets, QASR includes linguistically motivated segmentation, punctuation, speaker information, and more. The dataset is suitable for ASR, Arabic dialect identification, punctuation restoration, speaker identification, and NLP applications. Additionally, a 130M-word language model dataset is available to aid language modeling. Speech recognition models trained on QASR achieve competitive WER compared to the MGB-2 corpus, and it has been used for downstream tasks like Named Entity Recognition (NER) and punctuation restoration.

• ADI17 Dataset: ADI17 is a large-scale Arabic Dialect Identification (DID) dataset, collected from YouTube videos across 17 Arabic-speaking countries in the Middle East and North Africa. It contains 3,000 hours of speech for training DID systems and an additional 57 hours for development and testing. The dataset is categorized into short (<5s), medium (5-20s), and long (>20s) speech segments for detailed evaluation. ADI17 enables state-of-the-art dialect identification and provides a robust evaluation platform. It has been benchmarked on domain-mismatched conditions using the Multi-Genre Broadcast 3 (MGB-3) test set.