Learning Transferable Visual Models From Natural Language Supervision

Authors: Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, Ilya Sutskever (2021)

arXiv: 2103.00020

Domains

Multimodal

TLDR (English)

Uses 400M image-text pairs for contrastive learning to obtain universal vision encoder. CLIP embeddings remain the vision frontend for almost all multimodal systems (DALL·E, Stable Diffusion, LLaVA) today.

TLDR（中文）

用 4 亿对图文做对比学习，得到通用视觉 encoder。CLIP embedding 至今是几乎所有多模态系统（DALL·E、Stable Diffusion、LLaVA）的视觉前端。

Appears in These Articles

• RAG 与检索增强：让模型有外部记忆
• RAG and Retrieval Augmentation: Giving Models External Memory

Co-cited Papers

These papers appear in the same articles as this one

• Improving language models by retrieving from trillions of tokens Sebastian Borgeaud et al. (2022)
• Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao et al. (2022)
• Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin et al. (2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis et al. (2020)

Related Papers

Other papers in the same domain

• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  PretrainingMultimodal
• Visual Instruction Tuning Haotian Liu et al. (2023)
  Multimodal
• GPT-4V(ision) System Card OpenAI (2023)
  Multimodal
• Flamingo: a Visual Language Model for Few-Shot Learning Jean-Baptiste Alayrac et al. (2022)
  Multimodal
• Learning Transferable Visual Models From Natural Language Supervision Alec Radford et al. (2021)
  Multimodal