Visual Instruction Tuning

Authors: Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee (2023)

arXiv: 2304.08485

Domains

Multimodal

TLDR (English)

CLIP vision encoder + LLaMA + GPT-4 synthesized multimodal instruction data creates first open-source GPT-4V style model with minimal compute. Starting point for open-source multimodal ecosystem (LLaVA-1.5/1.6, Qwen-VL, InternVL).

TLDR（中文）

CLIP 视觉 encoder + LLaMA + GPT-4 合成的多模态指令数据，用极少算力做出第一个开源 GPT-4V 风格模型。开源多模态生态（LLaVA-1.5/1.6、Qwen-VL、InternVL）的范式起点。

Appears in These Articles

• RAG 与检索增强：让模型有外部记忆
• RAG and Retrieval Augmentation: Giving Models External Memory

Co-cited Papers

These papers appear in the same articles as this one

• Improving language models by retrieving from trillions of tokens Sebastian Borgeaud et al. (2022)
• Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao et al. (2022)
• Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin et al. (2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis et al. (2020)

Related Papers

Other papers in the same domain

• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  PretrainingMultimodal
• GPT-4V(ision) System Card OpenAI (2023)
  Multimodal
• Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac et al. (2022)
  Multimodal
• Learning Transferable Visual Models From Natural Language Supervision Alec Radford et al. (2021)
  Multimodal
• Learning Transferable Visual Models From Natural Language Supervision Alec Radford et al. (2021)
  Multimodal