Distributed Representations of Words and Phrases and their Compositionality

Authors: Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, Jeffrey Dean (2013)

arXiv: 1310.4546

Domains

Architecture

TLDR (English)

The NeurIPS version of word2vec, introducing Negative Sampling, Hierarchical Softmax, and phrase-level vectors. Influenced all subsequent embedding training objectives including GloVe, fastText, and modern LLM embedding layers.

TLDR（中文）

word2vec 的 NeurIPS 版正篇，引入 Negative Sampling、Hierarchical Softmax 与 phrase-level 向量。之后 GloVe / fastText / 大模型 embedding 层的训练目标都受其影响。

Appears in These Articles

• Embeddings：把离散符号放进连续空间
• Embeddings: Putting Discrete Symbols into Continuous Space

Co-cited Papers

These papers appear in the same articles as this one

• ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators Kevin Clark et al. (2020)
• Convolutional Neural Networks for Sentence Classification Yoon Kim (2014)
• Efficient Estimation of Word Representations in Vector Space Tomas Mikolov et al. (2013)
• GloVe: Global Vectors for Word Representation Jeffrey Pennington et al. (2014)

Related Papers

Other papers in the same domain

• DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI (2024)
  ArchitectureInference
• DeepSeek-V3 Technical Report DeepSeek-AI (2024)
  ArchitectureMixture of Experts
• Mixtral of Experts Albert Q. Jiang et al. (2024)
  Mixture of ExpertsArchitecture
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints Joshua Ainslie et al. (2023)
  InferenceArchitecture
• GLM-130B: An Open Bilingual Pre-trained Model Aohan Zeng et al. (2022)
  Architecture
• Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation Ofir Press et al. (2021)
  ArchitectureLong Context