Embedding

A relatively low-dimensional space into which high-dimensional vectors can be translated. Used in NLP to represent words or sentences.

How it works

An embedding layer maps discrete tokens (words, characters, or subwords) to dense continuous vectors. During training, these vectors are updated by backpropagation so that tokens that appear in similar contexts end up with similar vectors. In models like Word2Vec, embeddings are trained on a co-occurrence prediction task. In Transformers, token embeddings are learned jointly with the rest of the model and further modified by the attention layers to produce context-aware representations.

Why it matters

Embeddings are the bridge between discrete symbolic language and the continuous mathematics that neural networks operate in. They allow models to reason about semantic similarity — 'king' minus 'man' plus 'woman' approximately equals 'queen' in embedding space. They also power semantic search and RAG systems: by encoding documents and queries into the same embedding space, a vector database can retrieve semantically relevant content even when exact keywords don't match.