DeepSeek V3

DeepSeek V3 - Efficiency-First Sparse MoE with Architectural Co-Design

DeepSeek V3 is DeepSeek's flagship general-purpose model, built around the co-design of three documented architectural and training innovations: Multi-head Latent Attention (MLA) for inference efficiency, DeepSeekMoE, and a pioneering auxiliary-loss-free load balancing strategy. The full training proceeded with no irrecoverable loss spikes or rollbacks throughout.

Architecture

DeepSeek V3 is a Mixture-of-Experts model. Two core architectural components carry over from DeepSeek V2 and are the structural foundation:

• Multi-head Latent Attention (MLA): Compresses the KV cache into a low-dimensional latent space, enabling efficient long-context inference with manageable memory at 128K tokens
• DeepSeekMoE: The sparse MoE design that activates only a subset of parameters per token

Two additional strategies are introduced in V3:

• Auxiliary-loss-free load balancing: Pioneered in V3, this strategy balances expert utilization without adding a load-balancing loss term - eliminating the performance degradation that auxiliary loss terms typically cause
• Multi-Token Prediction (MTP): A training objective that predicts multiple future tokens per position simultaneously, documented as beneficial to model performance and usable for speculative decoding to accelerate inference.