Step 3.5 Flash

Step 3.5 Flash – Sparse MoE Agentic Reasoning Model

Step 3.5 Flash is StepFun's most capable open-source foundation model, built around the concept of intelligence density: a 196B-parameter sparse Mixture-of-Experts architecture that activates only ~11B parameters per token, targeting frontier-level reasoning at the latency profile required for real-time agentic interaction.

Architecture

The model uses a 45-layer sparse-MoE Transformer backbone (3 dense + 42 MoE layers). Each MoE layer contains 288 routed experts plus 1 always-active shared expert, with a top-8 router selecting active experts per token. This configuration preserves large-model knowledge capacity while constraining per-token compute.

Inference Design

Three co-designed mechanisms reduce wall-clock latency specifically for multi-turn agentic decoding:

• MTP-3 (3-way Multi-Token Prediction): Predicts 4 tokens per forward pass via a specialized head using sliding-window attention and a dense FFN, achieving 100–300 tok/s throughput (peaking at 350 tok/s on single-stream coding tasks)
• Hybrid Attention (3:1 SWA:Full ratio): Three sliding-window attention layers per full-attention layer, with window size 512, reducing long-context memory overhead
• Head-wise gated attention: Replaces fixed sink tokens with per-head gating, documented to raise pretraining benchmark averages.