BuboGPT

BuboGPT is a multi-modal large language model that extends LLM capabilities to incorporate image, audio, and text inputs with visual grounding. Developed by researchers at ByteDance, it uses a Q-Former and linear projection to connect modality encoders with the Vicuna language model, enabling a shared representation space. The model is trained in two stages: first, single-modal pre-training on large paired datasets; second, multi-modal instruction tuning using high-quality datasets including image-text, audio-text, and audio-image-text pairs with positive and negative examples. BuboGPT excels at fine-grained visual understanding, generating detailed audio descriptions, and cross-modal tasks such as sound localization and determining alignment between audio and image. It can handle both aligned and unaligned multi-modal inputs, providing informative responses with explicit grounding to visual objects.

Key Benefits

• Multi-modal input support (text, image, audio)
• Visual grounding capability
• Fine-grained understanding
• Handles aligned and unaligned multi-modal data