Solar Open2 250B

Solar Open2 250B - Hybrid-Attention MoE for Agentic Workflows

Solar Open2 250B is Upstage's open-weight large language model purpose-built for agentic use cases including office productivity, document-intensive work, and coding. Its defining architectural choice is a Hybrid-Attention Mixture-of-Experts design that interleaves three linear-attention layers with one softmax-attention layer in a repeating [Softmax, Linear×3] × 12 pattern across 48 layers.

The hybrid attention stack is central to the model's identity. By replacing three out of every four attention layers with linear attention, Solar Open2 250B achieves large-model capacity at a fraction of the inference cost. The removal of rotary positional encoding entirely (NoPE) is a direct consequence of the linear-attention design, enabling the 1M-token context window without the extrapolation limits typical of RoPE-based models.

The selective weight transfer from Solar Open 1 is another distinctive trait: rather than training from scratch, Upstage reused the small subset of weights compatible with the new architecture to raise the starting point and accelerate early convergence at 250B scale.