Improving language models by retrieving from trillions of tokens

作者： Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George van den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego de Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack W. Rae, Erich Elsen, Laurent Sifre (2022)

arXiv： 2112.04426

领域

应用

TLDR（中文）

DeepMind 在预训练阶段就引入 chunked retrieval，让 7B 模型匹敌 175B GPT-3。证明检索不只是 RAG 推理时招式，也是预训练范式的另一种可能。

TLDR (English)

DeepMind introduces chunked retrieval during pre-training, making 7B model match 175B GPT-3. Proves retrieval isn't just RAG inference trick, but another possible pre-training paradigm.

出现在这些文章里

• RAG 与检索增强：让模型有外部记忆
• RAG and Retrieval Augmentation: Giving Models External Memory

同被引用

这些论文与本文出现在同一篇文章中

• Precise Zero-Shot Dense Retrieval without Relevance Labels Luyu Gao et al. (2022)
• Dense Passage Retrieval for Open-Domain Question Answering Vladimir Karpukhin et al. (2020)
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks Patrick Lewis et al. (2020)
• Visual Instruction Tuning Haotian Liu et al. (2023)

相关论文

同一领域的其他论文

• Model Context Protocol (MCP) Anthropic (2024)
  应用
• SWE-bench: Can Language Models Resolve Real-World GitHub Issues? Carlos E. Jimenez et al. (2024)
  评估应用
• SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering John Yang et al. (2024)
  应用
• Toolformer: Language Models Can Teach Themselves to Use Tools Timo Schick et al. (2023)
  应用
• Reflexion: Language Agents with Verbal Reinforcement Learning Noah Shinn et al. (2023)
  应用
• Efficient Guided Generation for Large Language Models Brandon T. Willard et al. (2023)
  推理应用