Dimensionality Reduction

The transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties.

How it works

Techniques like Principal Component Analysis (PCA) find the directions of maximum variance in the data and project it onto fewer axes. t-SNE and UMAP take a different approach: they model the local neighbourhood structure of high-dimensional points and find a low-dimensional layout that preserves those neighbourhoods. Autoencoders use neural networks to learn a compressed latent code. Each technique balances the trade-off between preserving global structure, local structure, and computational cost.

Why it matters

High-dimensional data is computationally expensive to work with, prone to the curse of dimensionality, and impossible to visualise directly. Dimensionality reduction makes datasets manageable, improves the performance of downstream models by removing redundant or noisy features, and enables visualisation of complex data in 2D or 3D. It is fundamental to understanding what embeddings from language models actually capture about meaning and relationships.