Llama 4 Scout

Llama 4 Scout - Mixture-of-Experts Native Multimodal Model

Llama 4 Scout is Meta's open-weight, natively multimodal language model built on a mixture-of-experts (MoE) architecture with 16 experts. Despite a total of 109B parameters, only 17B parameters are active per token, enabling inference efficiency comparable to a dense 17B model. It is the first open-weight model from Meta to combine MoE with native multimodality via early fusion.

Architecture

Llama 4 Scout uses an auto-regressive MoE architecture with early fusion, integrating image and text inputs at the model level rather than through a separate vision encoder pipeline. This allows both modalities to share the same token space from the start of processing.

Context Strategy

Through mid-training with specialized long-context datasets, Scout achieves a 10 million token input context window — the largest in the Llama 4 series. This was extended using an interleaved RoPE (iRoPE) positional encoding approach to sustain quality across very long sequences.