Spaces:

GMFTBY

/

PandaGPT

like 80

Runtime error

Project Description

PandaGPT is the first foundation model capable of instruction-following data across six modalities, without the need of explicit supervision. It demonstrates a diverse set of multimodal capabilities such as complex understanding/reasoning, knowledge-grounded description, and multi-turn conversation.

PandaGPT is a general-purpose instruction-following model that can both see 👀 and hear👂. Our pilot experiments show that PandaGPT can perform complex tasks such as detailed image description generation, writing stories inspired by videos, and answering questions about audios. More Interestingly, PandaGPT can take multimodal inputs simultaneously and compose their semantics naturally. For example, PandaGPT can connect how objects look in a photo and how they sound in an audio.

Financial Situation

The major contributors of the project are mostly Ph.D. students. Unfortunately, due to financial constraints inherent to our status as students, self-funding such a critical resource is untenable. Therefore, we humbly request your consideration for the grant in question. With your support, we believe we can further contribute to the body of knowledge in our area of research, enhancing both our academic growth and potential societal impact.

Additional Links

For more details and use cases, please refer to our blog and github.