DeepSeek V3

DeepSeek V3 – Efficiency-First Sparse MoE with Architectural Co-Design

DeepSeek V3 is DeepSeek's flagship general-purpose model, built around the co-design of three documented architectural and training innovations: Multi-head Latent Attention (MLA) for inference efficiency, DeepSeekMoE for cost-effective training, and a pioneering auxiliary-loss-free load balancing strategy. The full training required 2.788M H800 GPU hours with no irrecoverable loss spikes or rollbacks throughout.

Architecture

DeepSeek V3 is a 671B total parameter, 37B active parameter Mixture-of-Experts model. Two core architectural components carry over from DeepSeek V2 and are the structural foundation:

• Multi-head Latent Attention (MLA): Compresses the KV cache into a low-dimensional latent space, enabling efficient long-context inference with manageable memory at 128K tokens
• DeepSeekMoE: The sparse MoE design enabling cost-effective training by activating only 37B of 671B parameters per token

Two additional strategies are introduced in V3:

• Auxiliary-loss-free load balancing: Pioneered in V3, this strategy balances expert utilization without adding a load-balancing loss term — eliminating the performance degradation that auxiliary loss terms typically cause
• Multi-Token Prediction (MTP): A training objective that predicts multiple future tokens per position simultaneously, documented as beneficial to model performance and usable for speculative decoding to accelerate inference.