Improving language models by retrieving from trillions of tokens

Authors: Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, Laurent Sifre (2022)

arXiv: 2112.04426

Domains

Applications

TLDR (English)

DeepMind introduces chunked retrieval during pre-training, making 7B model match 175B GPT-3. Proves retrieval isn't just RAG inference trick, but another possible pre-training paradigm.

TLDR（中文）

DeepMind 在预训练阶段就引入 chunked retrieval，让 7B 模型匹敌 175B GPT-3。证明检索不只是 RAG 推理时招式，也是预训练范式的另一种可能。

Appears in These Articles

• RAG 与检索增强：让模型有外部记忆
• RAG and Retrieval Augmentation: Giving Models External Memory

Co-cited Papers

These papers appear in the same articles as this one

• Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao et al. (2022)
• Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin et al. (2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis et al. (2020)
• Visual Instruction Tuning Haotian Liu et al. (2023)

Related Papers

Other papers in the same domain

• Model Context Protocol (MCP) Anthropic (2024)
  Applications
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  EvaluationApplications
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering John Yang et al. (2024)
  Applications
• Toolformer: Language Models Can Teach Themselves to Use Tools Timo Schick et al. (2023)
  Applications
• Reflexion: Language Agents with Verbal Reinforcement Learning Noah Shinn et al. (2023)
  Applications
• Efficient Guided Generation for Large Language Models Brandon T. Willard et al. (2023)
  InferenceApplications