Glossary
AI & ML Glossary
Explore our extensive glossary of terms related to Artificial Intelligence, Machine Learning, and Large Language Models.
A* Algorithm
A popular pathfinding and graph traversal algorithm that is highly favored for its performance and accuracy.
Activation Function
A mathematical function applied to a neural network node to help it learn complex patterns.
AGI (Artificial General Intelligence)
A hypothetical type of AI that can understand, learn, and apply knowledge across a wide range of tasks at a level equal to or beyond human capabilities.
Algorithm
A set of rules or instructions given to an AI, neural network, or other machines to help them learn on their own.
Alignment
The process of ensuring AI systems are aligned with human values and goals.
Simulated Annealing
A probabilistic technique for approximating the global optimum of a given function, inspired by metallurgy.
Anthropic
An AI safety and research company that builds models like Claude.
API (Application Programming Interface)
A way for two or more computer programs to communicate with each other. Often used to integrate AI models into applications.
Artificial Intelligence (AI)
The simulation of human intelligence processes by machines, especially computer systems.
Attention Mechanism
A technique in neural networks that allows the model to focus on specific parts of the input when generating an output, heavily used in Transformers.
Attention Sink
A pattern where early tokens (often the system prompt) keep receiving disproportionate attention across a long generation, helping the model stay anchored to core instructions.
Auto-GPT
An experimental open-source application showcasing the capabilities of the GPT-4 language model. It chains together LLM thoughts to autonomously achieve whatever goal you set.
Autoencoder
A type of artificial neural network used to learn efficient codings of unlabeled data (unsupervised learning).
Backpropagation
A method used in artificial neural networks to calculate the error contribution of each neuron after a batch of data is processed.
Batch Normalization
A technique used to make training artificial neural networks faster and more stable by normalizing the inputs of the layers.
Bayesian Network
A probabilistic graphical model that represents a set of variables and their conditional dependencies via a directed acyclic graph.
Best-of-N Sampling
An inference technique that generates N full candidate responses, scores them with a reward or preference model, and returns only the highest-scoring one.
Bias (in AI)
Systematic and repeatable errors in a computer system that create unfair outcomes, such as privileging one arbitrary group of users over others.
Character.ai
A consumer AI entertainment product known for long, character-based chats. Its engineering stack (including the Kaiju model family, memory systems, and Lorebook) is documented in public Character.ai blog posts.
ChatGPT
A conversational AI interface developed by OpenAI, based on the GPT family of large language models.
Claude
A family of foundational AI models created by Anthropic, designed to be helpful, harmless, and honest.
Clustering
The task of grouping a set of objects in such a way that objects in the same group are more similar to each other than to those in other groups.
Computer Vision
A field of AI that trains computers to interpret and understand the visual world.
Constitutional AI
An alignment approach (popularised by Anthropic) where a model critiques and revises its own outputs against a written set of principles, instead of relying only on human preference labels.
Context Window
The amount of text (measured in tokens) that a large language model can consider at one time when generating a response.
Convolutional Neural Network (CNN)
A class of artificial neural network commonly used to analyze visual imagery.
Cosine Similarity
A metric used to measure how similar two vectors are, commonly used in NLP and vector databases.
Data Mining
The practice of analyzing large databases in order to generate new information.
Decision Tree
A decision support tool that uses a tree-like model of decisions and their possible consequences.
Deep Learning
A subset of machine learning based on artificial neural networks with multiple layers.
DeepSeek
A Chinese AI lab and model family known for open technical reports on MoE transformers and Multi-Head Latent Attention (MLA), including DeepSeek-V2, V3, and related chat / reasoning assistants.
DeepSeek-R1
DeepSeek’s reasoning model trained with reinforcement learning to produce long, often visible chain-of-thought traces before the final answer.
DeepSeek-V2
Earlier DeepSeek MoE model (about 236B total / 21B active, 128K context) that introduced Multi-Head Latent Attention (MLA) as a headline feature (arXiv:2405.04434).
DeepSeek-V3
DeepSeek’s MoE language model with about 671B total parameters and 37B activated per token, known for MLA, fine-grained experts, and auxiliary-loss-free load balancing (arXiv:2412.19437).
Diffusion Model
A class of generative models that learns to denoise images, commonly used for high-quality image generation.
Dimensionality Reduction
The transformation of data from a high-dimensional space into a low-dimensional space so that the low-dimensional representation retains some meaningful properties.
Dropout
A regularization technique for reducing overfitting in neural networks by preventing complex co-adaptations on training data.
Embedding
A relatively low-dimensional space into which high-dimensional vectors can be translated. Used in NLP to represent words or sentences.
Ensemble Learning
The process by which multiple models, such as classifiers or experts, are strategically generated and combined to solve a particular computational intelligence program.
EOS (End of Sequence)
A special token that signals the language model to stop generating further tokens for the current response.
Epoch
One complete pass of the training dataset through the algorithm during the machine learning training process.
Expert System
A computer system that emulates the decision-making ability of a human expert.
Explainable AI (XAI)
Artificial intelligence in which the results of the solution can be understood by humans.
Few-shot Learning
The practice of feeding a machine learning model with a very small amount of training data, contrary to the normal practice of using a large amount.
Fine-tuning
Taking a pre-trained model and training it further on a specific dataset to adapt it to a specialized task.
FP8 Mixed Precision
Training or inference that uses 8-bit floating-point formats for much of the compute, cutting memory and speeding up large model training when done carefully.
Generative AI
A type of artificial intelligence that can generate new content, such as text, imagery, audio, and synthetic data.
Generative Adversarial Network (GAN)
A class of machine learning frameworks designed to generate new data with the same statistics as the training set, consisting of a generator and a discriminator.
Gradient Descent
An optimization algorithm used for minimizing the cost function in various machine learning algorithms.
GPT (Generative Pre-trained Transformer)
A series of large language models developed by OpenAI.
Hallucination
When an AI model generates false, fabricated, or nonsensical information but presents it as a fact.
Heuristic
A technique designed for solving a problem more quickly when classic methods are too slow, or for finding an approximate solution when classic methods fail.
Hyperparameter
A parameter whose value is used to control the learning process of a machine learning algorithm.
Inference
The process of using a trained machine learning model to make a prediction or generate text.
K-Means Clustering
A method of vector quantization that aims to partition n observations into k clusters in which each observation belongs to the cluster with the nearest mean.
Knowledge Cutoff
The last date covered by a model’s training data. After that date the model has no reliable built-in facts unless tools or retrieval supply them.
Knowledge Graph
A knowledge base that uses a graph-structured data model or topology to integrate data.
Large Language Model (LLM)
An AI model trained on vast amounts of text data, capable of understanding and generating human-like text.
LayerNorm (Layer Normalization)
A normalization step inside Transformer blocks that stabilises activations (often applied before each sublayer in modern Pre-LN designs).
Latent Space
An abstract multi-dimensional space containing feature values that a machine learning model cannot interpret directly but uses to understand relationships.
LLaMA
A family of large language models released by Meta.
Machine Learning (ML)
A subset of AI that involves training algorithms to learn from data and make predictions.
MCP (Model Context Protocol)
A standard that enables AI models to interact with external tools and context seamlessly.
Midjourney
An independent research lab and generative AI program that creates images from text descriptions.
Model Collapse
A phenomenon where an AI model trained heavily on AI-generated data begins to degrade in quality and lose its representation of the real-world distribution.
Multi-Token Prediction
A training objective where the model predicts several future tokens at once to improve representations; at normal chat inference it still usually emits one token at a time.
Natural Language Generation (NLG)
The use of artificial intelligence programming to produce written or spoken narratives from a dataset.
NLP (Natural Language Processing)
A branch of AI that helps computers understand, interpret, and manipulate human language.
Neural Network
A series of algorithms that endeavors to recognize underlying relationships in a set of data through a process that mimics the way the human brain operates.
OpenAI
An artificial intelligence research laboratory and company, creators of ChatGPT and DALL-E.
Overfitting
A modeling error which occurs when a function is too closely fit to a limited set of data points, performing poorly on new data.
Parameters
The variables in an artificial neural network that are adjusted during training to minimize the error in the model's predictions.
Prefill and Decode
Two phases of Transformer inference: prefill processes the full prompt (often in parallel), then decode generates new tokens one by one using the KV cache.
Predictive Analytics
The use of statistical techniques and machine learning to predict future outcomes based on historical data.
Prompt Engineering
The process of designing and refining inputs to generative AI models to produce optimal outputs.
RAG (Retrieval-Augmented Generation)
An AI framework that improves the quality of LLM-generated responses by grounding the model on external sources of knowledge.
Random Forest
An ensemble learning method for classification, regression and other tasks that operates by constructing a multitude of decision trees.
Recurrent Neural Network (RNN)
A class of artificial neural networks where connections between nodes form a directed graph along a temporal sequence, good for sequential data.
Reinforcement Learning (RL)
An area of machine learning concerned with how software agents ought to take actions in an environment in order to maximize the notion of cumulative reward.
Repetition Penalty
A decoding adjustment that lowers the chance of recently used tokens so the model is less likely to loop or stall on the same phrases.
Residual Connection
A skip connection that adds a sublayer’s input to its output, helping deep Transformers train stably by letting gradients flow through the network.
RLHF (Reinforcement Learning from Human Feedback)
A technique used to train AI models by incorporating human feedback to reward desired behaviors.
Robotics
The intersection of science, engineering and technology that produces machines, called robots, that substitute for (or replicate) human actions.
Self-Supervised Learning
A machine learning process where the model generates its own labels from the input data, often used in large language models.
Sentiment Analysis
The use of natural language processing to systematically identify, extract, quantify, and study affective states and subjective information.
SFT (Supervised Fine-Tuning)
Fine-tuning a base model on curated instruction–response pairs so it learns to act like a helpful assistant rather than only continuing raw text.
Softmax
A function that turns a vector of raw scores (logits) into a probability distribution that sums to one, used when sampling the next token.
Speculative Decoding
A speed-up where a smaller draft model proposes several tokens and the main model verifies them in parallel, accepting or rejecting the draft.
Stable Diffusion
A deep learning, text-to-image model released in 2022 by Stability AI.
Superintelligence
A hypothetical agent that possesses intelligence far surpassing that of the brightest and most gifted human minds.
Supervised Learning
The machine learning task of learning a function that maps an input to an output based on example input-output pairs.
Support Vector Machine (SVM)
Supervised learning models with associated learning algorithms that analyze data for classification and regression analysis.
Swarm Intelligence
The collective behavior of decentralized, self-organized systems, natural or artificial.
Temperature
A parameter in language models that controls the randomness or creativity of the output. Higher values lead to more random outputs.
Tensor
A mathematical object analogous to but more general than a vector, widely used in machine learning as multi-dimensional data arrays.
TensorFlow
A free and open-source software library for machine learning and artificial intelligence, developed by Google.
Token
The basic unit of text that an LLM processes. A token can be a word, part of a word, or just a character.
Transformer
A deep learning architecture introduced in 2017 that relies entirely on an attention mechanism to draw global dependencies between input and output.
Turing Test
A test of a machine's ability to exhibit intelligent behavior equivalent to, or indistinguishable from, that of a human.
Underfitting
When a machine learning model cannot adequately capture the underlying structure of the data.
Unsupervised Learning
A type of machine learning that looks for previously undetected patterns in a data set with no pre-existing labels.
Vector Database
A database specifically designed to store and query high-dimensional vectors, often used for embeddings in RAG systems.
Weights
In a neural network, weights decide how much influence the input will have on the output.
YaRN
A method for extending Rotary Position Embeddings so a model trained on shorter sequences can handle much longer contexts at inference time.
Zero-shot Learning
The ability of an AI model to complete a task without having received any specific training examples for that task.
AI Agent
An autonomous system driven by AI that can perceive its environment, make decisions, and take actions to achieve specific goals.
Autonomous Agent
An AI system capable of independently acting and making decisions in an environment to achieve its programmed goals without human intervention.
LoRA (Low-Rank Adaptation)
A highly efficient fine-tuning technique that drastically reduces the number of trainable parameters by injecting small rank-decomposition matrices into each layer of the Transformer architecture.
Quantization
The process of reducing the precision of the numbers used to represent a model's parameters (e.g., from 32-bit floats to 8-bit or 4-bit integers) to reduce memory usage and improve inference speed.
MoE (Mixture of Experts)
A machine learning technique where only a subset of a neural network's parameters (the 'experts') are activated for any given input, allowing for massive model scale without a proportional increase in compute cost.
KV Cache (Key-Value Cache)
A memory optimization technique used during Transformer inference to store previously computed key and value vectors, avoiding redundant calculations for past tokens.
Chain of Thought (CoT)
A prompting strategy that encourages an LLM to generate a step-by-step reasoning process before providing its final answer, significantly improving performance on complex tasks.
DPO (Direct Preference Optimization)
A streamlined alternative to RLHF that directly optimizes a language model to align with human preferences without needing a separate reward model.
Instruction Tuning
The process of fine-tuning a base language model on a dataset of instructions and expected responses so it learns to follow user commands rather than just predicting the next word.
System Prompt
A special set of instructions provided to an LLM at the very beginning of a conversation to dictate its persona, rules, and constraints.
RoPE (Rotary Position Embedding)
A mechanism used in modern Transformers to inject positional information into tokens, allowing models to better handle relative distances and extrapolate to longer sequence lengths.
FlashAttention
An I/O-aware, exact attention algorithm that minimizes memory reads/writes to the GPU, making training and inference of Transformers significantly faster and more memory-efficient.
Tokenization
The process of breaking down raw text into smaller pieces (tokens) such as words, subwords, or characters, so they can be processed by a neural network.
PPO (Proximal Policy Optimization)
A popular reinforcement learning algorithm used in the RLHF phase of training large language models to balance stability and training speed.
RLAIF (Reinforcement Learning from AI Feedback)
A scalable alternative to RLHF where an AI model, rather than human annotators, is used to evaluate and rank outputs to train a reward model.
PEFT (Parameter-Efficient Fine-Tuning)
A family of techniques (including LoRA) designed to fine-tune large pre-trained models by only updating a very small number of extra parameters, saving vast amounts of compute and memory.
GGUF
A binary file format used for storing models for inference (especially via llama.cpp), allowing them to run efficiently on standard consumer hardware, including CPUs.
VRAM (Video RAM)
The memory located on a graphics card (GPU). In AI, sufficient VRAM is critical for loading large models and their context windows entirely into GPU memory for fast inference.
Top-P (Nucleus Sampling)
A decoding method where the model only considers the smallest set of tokens whose cumulative probability exceeds the probability P, allowing for dynamic vocabulary selection.
Top-K Sampling
A decoding strategy that restricts the AI model to only sample from the K most likely next tokens, cutting off long-tail probabilities to reduce nonsensical outputs.
Beam Search
A heuristic search algorithm that explores a graph by expanding the most promising nodes in a limited set (the 'beam'), often used to find the most likely sentence translation or output sequence.
Perplexity
A measurement of how well a probability distribution or language model predicts a sample. Lower perplexity indicates the model is better at predicting the sequence.
BPE (Byte-Pair Encoding)
A subword tokenization algorithm that iteratively merges the most frequently occurring character or byte pairs, widely used by models like GPT to handle out-of-vocabulary words.
Causal Attention
An attention mask used in autoregressive language models so each token can only attend to itself and earlier tokens, never to future tokens.
Cross-Encoder
A neural network architecture where two input texts are concatenated and processed together, yielding highly accurate similarity scores at the cost of high computational overhead.
Bi-Encoder
An architecture where two input texts are encoded separately into dense vectors and then compared (e.g., using cosine similarity), which is much faster than cross-encoders and powers vector databases.
Agent Swarm
A multi-agent setup where one orchestrator model spins up many specialized sub-agents that work in parallel on parts of a large task (as described for Moonshot Kimi K2.6).
Autoregressive
A generation style where each new token is predicted from all previous tokens only. Future tokens cannot change earlier ones.
Feed-Forward Network (FFN)
The fully connected layers inside each Transformer block that transform attention outputs. In MoE models, these FFNs are often the routed experts.
SwiGLU
A gated activation function used in many modern LLM feed-forward blocks: SiLU on one projection multiplied by another linear gate, giving stronger expressiveness than a single linear layer plus ReLU.
Kimi
An AI assistant and open-weight model family built by Moonshot AI, known for long-context chat and agentic tool use (including Kimi K2 / K2.6).
Logits
Raw, unnormalized scores a model assigns to each possible next token before softmax turns them into probabilities.
MLA (Multi-Head Latent Attention)
An attention design that compresses key/value caches into a smaller latent vector to cut memory use on long contexts, used in models such as DeepSeek-V3 and Kimi K2.
Moonshot AI
A Beijing-based AI company (月之暗面) that builds the Kimi assistant and open-weight models such as Kimi K2 / K2.6.
Muon Optimizer
A training optimizer used in Moonshot’s Kimi K2 work that aims to get more model quality from the same token budget than a plain AdamW-style setup.
Open Weights
Model weights released so others can download, run, and often fine-tune them locally or on their own cloud, usually under a permissive or research license.
QK-Clip
A training stability technique from Moonshot’s Kimi K2 report that clips extreme query-key attention scores to keep long-context training stable.
Ring Attention
A distributed attention method that shards a long sequence across devices in a ring so models can handle much longer contexts than a single GPU’s memory allows.
Sparse Attention
An attention pattern that only looks at a subset of tokens instead of every pair, reducing the quadratic cost of full attention on long sequences.
Tool Calling
When an LLM emits structured calls to external tools (search, code, APIs) and then continues generation after the tool results return.
bfloat16 (BF16)
A 16-bit floating-point format widely used in deep learning. It keeps the same exponent range as FP32 so training stays stable, while using less memory than full precision.
FSDP (Fully Sharded Data Parallel)
A distributed training method that shards model parameters, gradients, and optimiser state across GPUs so very large models can train without fitting the full copy on every device.
INT8
8-bit integer precision. Using INT8 for weights, activations, or KV values cuts memory and can roughly double matrix-multiply throughput on many accelerators versus 16-bit floats.
Kaiju
Character.ai’s in-house family of dense transformer LLMs (Small 13B, Medium 34B, Large 110B), built for fast, engaging, safety-aware conversation rather than raw academic benchmarks alone.
Kubernetes
An open-source system for running and healing containerised workloads across a cluster — the usual backbone for modern cloud training and serving platforms.
Lorebook
A Character.ai world-building feature: reusable keyword-triggered entries (locations, side characters, rules, items) that inject into chat only when relevant keywords appear.
Model Parallelism
Splitting a single large model across devices (for example tensor, pipeline, sequence, or expert parallelism) so training or inference can exceed one GPU’s memory.
Multi-Query Attention (MQA)
An attention design that shares Key and Value projections across attention heads, shrinking the KV cache and speeding up chat inference compared with full multi-head attention.
Pretraining Annealing
Near the end of pretraining, gradually shifting the data mix toward higher-quality or instruction-style examples so the model settles into better behaviour before fine-tuning.
Quantization-Aware Training (QAT)
Training a model while simulating lower-precision arithmetic (for example INT8) so weights learn to stay accurate after quantization, instead of only quantizing after training.
RMSNorm
Root Mean Square Layer Normalization — a lighter normalisation used in many modern Transformers. It scales activations by their RMS without subtracting the mean like classic LayerNorm.
Sliding Window Attention
An attention pattern where most layers only attend to a recent window of tokens (for example 1024), often interleaved with occasional global-attention layers to keep long-context ability cheaper.
Slurm
A widely used HPC job scheduler (sbatch, squeue, fair-share, preemption). Researchers often prefer its UX even when the cluster underneath runs on Kubernetes.
Squinch
A 6-bit block-wise gradient compression algorithm invented by Noam Shazeer at Character.ai to cut multi-node communication bandwidth during large-scale pretraining while limiting reconstruction error.
Sticky Session
Routing successive requests from the same user or dialogue to the same server so cached state (such as a KV cache) can be reused and hit rates stay high.
TTFT (Time To First Token)
Latency from sending a request until the first generated token arrives. A key user-facing metric for chat apps — how quickly the reply starts streaming.
TPOT (Time Per Output Token)
Average time between successive generated tokens after the first one. Together with TTFT, it describes how fast a chat feels once streaming has begun.
vLLM
A high-throughput open-source LLM serving engine known for efficient continuous batching and KV-cache management in production inference.