Language Models are Unsupervised Multitask Learners

作者： Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever (2019)

领域

预训练

TLDR（中文）

GPT-2 展示了一个仅在未标注网络文本上训练的 15 亿参数语言模型，能在无任何微调的情况下 以零样本方式完成多种语言任务。这挑战了"NLP 任务必须任务专属训练"的传统观念， 也因担心被滥用而成为第一个"延迟发布"的 AI 模型。

TLDR (English)

GPT-2 shows that a 1.5B parameter language model trained only on unlabeled web text can perform various language tasks zero-shot without fine-tuning. This challenged the convention that NLP tasks require task-specific training and famously became the first AI model "staged released" due to misuse concerns.

出现在这些文章里

• Transformer Architecture：现代 LLM 的骨架
• Transformer Architecture: The Skeleton of Modern LLMs

同被引用

这些论文与本文出现在同一篇文章中

• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Jacob Devlin et al. (2018)
• GLaM: Efficient Scaling of Language Models with Mixture-of-Experts Nan Du et al. (2021)
• Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity William Fedus et al. (2021)
• Mixtral of Experts Albert Q. Jiang et al. (2024)

相关论文

同一领域的其他论文

• Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone Microsoft (2024)
  预训练
• Yi: Open Foundation Models by 01.AI 01. AI et al. (2024)
  预训练
• The Llama 3 Herd of Models Meta AI (2024)
  预训练
• Qwen2.5 Technical Report Qwen et al. (2024)
  预训练
• Qwen Technical Report Jinze Bai et al. (2023)
  预训练
• Gemini: A Family of Highly Capable Multimodal Models Gemini Team et al. (2023)
  预训练多模态