Visual Instruction Tuning

作者： Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee (2023)

arXiv： 2304.08485

领域

多模态

TLDR（中文）

CLIP 视觉 encoder + LLaMA + GPT-4 合成的多模态指令数据，用极少算力做出第一个开源 GPT-4V 风格模型。开源多模态生态（LLaVA-1.5/1.6、Qwen-VL、InternVL）的范式起点。

TLDR (English)

CLIP vision encoder + LLaMA + GPT-4 synthesized multimodal instruction data creates first open-source GPT-4V style model with minimal compute. Starting point for open-source multimodal ecosystem (LLaVA-1.5/1.6, Qwen-VL, InternVL).

出现在这些文章里

• RAG 与检索增强：让模型有外部记忆
• RAG and Retrieval Augmentation: Giving Models External Memory

同被引用

这些论文与本文出现在同一篇文章中

• Improving language models by retrieving from trillions of tokens Sebastian Borgeaud et al. (2022)
• Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao et al. (2022)
• Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin et al. (2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis et al. (2020)

相关论文

同一领域的其他论文

• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  预训练多模态
• GPT-4V(ision) System Card OpenAI (2023)
  多模态
• Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac et al. (2022)
  多模态
• Learning Transferable Visual Models From Natural Language Supervision Alec Radford et al. (2021)
  多模态
• Learning Transferable Visual Models From Natural Language Supervision Alec Radford et al. (2021)
  多模态