Flowing from Words to Pixels: A Framework for Cross-Modality Evolution
Paper
•
2412.15213
•
Published
•
25
Paper - Multimodal
Paper related to Multimodal Model - Research for a : Modular, Multimodal, Multi-Stream, Mixture of Expert, Universal Transformer, Matryoshka embedding
-
-
No More Adam: Learning Rate Scaling at Initialization is All You Need
Paper • 2412.11768 • Published • 41 -
Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inference
Paper • 2412.13663 • Published • 103 -
Autoregressive Video Generation without Vector Quantization
Paper • 2412.14169 • Published • 13 -
ChatDiT: A Training-Free Baseline for Task-Agnostic Free-Form Chatting with Diffusion Transformers
Paper • 2412.12571 • Published • 8 -
Byte Latent Transformer: Patches Scale Better Than Tokens
Paper • 2412.09871 • Published • 74 -
Causal Diffusion Transformers for Generative Modeling
Paper • 2412.12095 • Published • 23 -
Apollo: An Exploration of Video Understanding in Large Multimodal Models
Paper • 2412.10360 • Published • 131 -
Large Action Models: From Inception to Implementation
Paper • 2412.10047 • Published • 28 -
Multimodal Music Generation with Explicit Bridges and Retrieval Augmentation
Paper • 2412.09428 • Published • 7 -
Phi-4 Technical Report
Paper • 2412.08905 • Published • 92 -
MetaMorph: Multimodal Understanding and Generation via Instruction Tuning
Paper • 2412.14164 • Published • 1 -
InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for Long-term Streaming Video and Audio Interactions
Paper • 2412.09596 • Published • 90 -
OLA-VLM: Elevating Visual Perception in Multimodal LLMs with Auxiliary Embedding Distillation
Paper • 2412.09585 • Published • 10 -
Meshtron: High-Fidelity, Artist-Like 3D Mesh Generation at Scale
Paper • 2412.09548 • Published • 3 -
Multimodal Latent Language Modeling with Next-Token Diffusion
Paper • 2412.08635 • Published • 41 -
MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation
Paper • 2412.07147 • Published • 5 -
Moshi: a speech-text foundation model for real-time dialogue
Paper • 2410.00037 • Published • 1 -
Training Large Language Models to Reason in a Continuous Latent Space
Paper • 2412.06769 • Published • 62 -
Marco-o1: Towards Open Reasoning Models for Open-Ended Solutions
Paper • 2411.14405 • Published • 58 -
O1 Replication Journey: A Strategic Progress Report -- Part 1
Paper • 2410.18982 • Published • 1 -
O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?
Paper • 2411.16489 • Published • 40 -
The Llama 3 Herd of Models
Paper • 2407.21783 • Published • 109 -
PaliGemma 2: A Family of Versatile VLMs for Transfer
Paper • 2412.03555 • Published • 118 -
Reverse Thinking Makes LLMs Stronger Reasoners
Paper • 2411.19865 • Published • 19 -
Differential Transformer
Paper • 2410.05258 • Published • 168 -
Large Concept Models: Language Modeling in a Sentence Representation Space
Paper • 2412.08821 • Published • 7 -
AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modeling
Paper • 2412.15084 • Published • 12 -
Mixtral of Experts
Paper • 2401.04088 • Published • 158 -
Pixtral 12B
Paper • 2410.07073 • Published • 62 -
A Generalist Agent
Paper • 2205.06175 • Published • 4 -
Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agent
Paper • 2402.09844 • Published • 20 -
A Comprehensive Overview of Large Language Models
Paper • 2307.06435 • Published • 2 -
RoFormer: Enhanced Transformer with Rotary Position Embedding
Paper • 2104.09864 • Published • 11 -
Rho-1: Not All Tokens Are What You Need
Paper • 2404.07965 • Published • 87 -
Revisiting Token Dropping Strategy in Efficient BERT Pretraining
Paper • 2305.15273 • Published • 1 -
Taming Multimodal Joint Training for High-Quality Video-to-Audio Synthesis
Paper • 2412.15322 • Published • 15 -
Cut Your Losses in Large-Vocabulary Language Models
Paper • 2411.09009 • Published • 43 -
Parallelized Autoregressive Visual Generation
Paper • 2412.15119 • Published • 44 -
Large Language Models can Learn Rules
Paper • 2310.07064 • Published • 1 -
Hypothesis Search: Inductive Reasoning with Language Models
Paper • 2309.05660 • Published • 2 -
Multimodal Pathway: Improve Transformers with Irrelevant Data from Other Modalities
Paper • 2401.14405 • Published • 12 -
Janus: Decoupling Visual Encoding for Unified Multimodal Understanding and Generation
Paper • 2410.13848 • Published • 31 -
MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
Paper • 2409.20566 • Published • 53 -
Emu3: Next-Token Prediction is All You Need
Paper • 2409.18869 • Published • 93 -
Florence-VL: Enhancing Vision-Language Models with Generative Vision Encoder and Depth-Breadth Fusion
Paper • 2412.04424 • Published • 55 -
MegaPairs: Massive Data Synthesis For Universal Multimodal Retrieval
Paper • 2412.14475 • Published • 51 -
VisionZip: Longer is Better but Not Necessary in Vision Language Models
Paper • 2412.04467 • Published • 104 -
Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoders
Paper • 2408.15998 • Published • 84 -
MIO: A Foundation Model on Multimodal Tokens
Paper • 2409.17692 • Published • 52 -
Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models
Paper • 2403.18814 • Published • 45 -
Matryoshka Multimodal Models
Paper • 2405.17430 • Published • 31 -
Needle In A Multimodal Haystack
Paper • 2406.07230 • Published • 53 -
Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMs
Paper • 2406.16860 • Published • 59 -
COSMO: COntrastive Streamlined MultimOdal Model with Interleaved Pre-Training
Paper • 2401.00849 • Published • 15 -
Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Tasks
Paper • 2206.08916 • Published • 1 -
Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Action
Paper • 2312.17172 • Published • 27 -
ImageBind: One Embedding Space To Bind Them All
Paper • 2305.05665 • Published • 5 -
Dense Connector for MLLMs
Paper • 2405.13800 • Published • 22 -
MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training
Paper • 2403.09611 • Published • 125 -
VisionArena: 230K Real World User-VLM Conversations with Preference Labels
Paper • 2412.08687 • Published • 13 -
LAION-SG: An Enhanced Large-Scale Dataset for Training Complex Image-Text Models with Structural Annotations
Paper • 2412.08580 • Published • 44 -
Maya: An Instruction Finetuned Multilingual Multimodal Model
Paper • 2412.07112 • Published • 25 -
OmniFlow: Any-to-Any Generation with Multi-Modal Rectified Flows
Paper • 2412.01169 • Published • 11 -
SoundStorm: Efficient Parallel Audio Generation
Paper • 2305.09636 • Published • 4 -
SoundStream: An End-to-End Neural Audio Codec
Paper • 2107.03312 • Published -
Descriptive Caption Enhancement with Visual Specialists for Multimodal Perception
Paper • 2412.14233 • Published • 6 -
Offline Reinforcement Learning for LLM Multi-Step Reasoning
Paper • 2412.16145 • Published • 30 -
Progressive Multimodal Reasoning via Active Retrieval
Paper • 2412.14835 • Published • 66 -
Vision-Language Models for Vision Tasks: A Survey
Paper • 2304.00685 • Published • 1 -
An Introduction to Vision-Language Modeling
Paper • 2405.17247 • Published • 87 -
Efficient Large Language Models: A Survey
Paper • 2312.03863 • Published • 3 -
A Survey on Multimodal Large Language Models
Paper • 2306.13549 • Published • 1 -
A Survey on Benchmarks of Multimodal Large Language Models
Paper • 2408.08632 • Published • 1 -
Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoning
Paper • 2401.06805 • Published • 2 -
Speech is More Than Words: Do Speech-to-Text Translation Systems Leverage Prosody?
Paper • 2410.24019 • Published • 1 -
Multimodal Autoregressive Pre-training of Large Vision Encoders
Paper • 2411.14402 • Published • 43 -
4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities
Paper • 2406.09406 • Published • 14 -
Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculum
Paper • 2405.13226 • Published • 1 -
Kaleido Diffusion: Improving Conditional Diffusion Models with Autoregressive Latent Modeling
Paper • 2405.21048 • Published • 13 -
Towards Time Series Reasoning with LLMs
Paper • 2409.11376 • Published • 1 -
Computational Bottlenecks of Training Small-scale Large Language Models
Paper • 2410.19456 • Published • 1 -
Depth Pro: Sharp Monocular Metric Depth in Less Than a Second
Paper • 2410.02073 • Published • 41 -
Contrastive Localized Language-Image Pre-Training
Paper • 2410.02746 • Published • 33 -
Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs
Paper • 2404.05719 • Published • 82 -
Optimizing Byte-level Representation for End-to-end ASR
Paper • 2406.09676 • Published • 1 -
Can you Remove the Downstream Model for Speaker Recognition with Self-Supervised Speech Features?
Paper • 2402.00340 • Published • 1 -
A Touch, Vision, and Language Dataset for Multimodal Alignment
Paper • 2402.13232 • Published • 14 -
MoEUT: Mixture-of-Experts Universal Transformers
Paper • 2405.16039 • Published • 1 -
Are Transformers with One Layer Self-Attention Using Low-Rank Weight Matrices Universal Approximators?
Paper • 2307.14023 • Published • 1 -
Deliberation in Latent Space via Differentiable Cache Augmentation
Paper • 2412.17747 • Published • 20 -
Vision Transformers Need Registers
Paper • 2309.16588 • Published • 78