Llama 4: The Natively Multimodal AI

Today Meta AI announced the release of Llama 4, newest iteration of their Llama series.

Llama 4 is their first natively multimodal AI that can process text, images and videos in a single model. This means it can understand and generate content in multiple formats, making it more versatile and capable than previous models.

Llama 4 is a mixture of experts (MoE) model, which means it uses a combination of different expert models to process different types of data. The model can switch between different experts based on the type of data it is processing, allowing it to adapt to different tasks and environments.

This Llama 4 release consists of 3 models

• Llama 4: Scout
  • Active Parameters: 17B
  • Expert models: 16
  • Total Parameters: 109B
  • Context Length: 10M Tokens
• Llama 4: Maverick
  • Active Parameters: 17B
  • Expert models: 128
  • Total Parameters: 400B
  • Context Length: 1M Tokens
• Llama 4: Behemoth
  • Active Parameters: 288B
  • Expert models: 16
  • Total Parameters: 2T
  • The model is still under training
  • Behemoth serves as a powerful "teacher" model, distilling its knowledge into the smaller Llama 4 variants

Architecture

The Llama 4 architecture introduces key innovations, most notably theMixture-of-Experts (MoE) design, where different specialized "expert" models are activated depending on the input, for better efficiency and capability. For handling long sequences, Llama 4 Scout employs the novel iRoPE architecture, featuring interleaved attention layers without positional embeddings and inference time temperature scaling to improve length generalization, building upon rotary position embeddings (RoPE) used in most layers. This focus on MoE and advanced attention mechanisms defines the core architectural advancements in the Llama 4 family.

Mixture of Experts (MoE) architecture (Source: Meta AI Blog)

The remarkable capabilities of Llama 4 are built upon significant advancements in both pre-training and post-training methodologies.

Training Data

Llama 4 also boasts significantly enhanced multilingual capabilities, being pre-trained on 200 languages with a massive increase in multilingual tokens compared to Llama 3. The sheer scale of the pre-training data is behemoth (pun intended), exceeding 30 trillion tokens.

Pre-training details

During pre-training, Meta AI adopted a native multimodality approach, enabling the models to learn from vast datasets of unlabeled text, images, and videos simultaneously through early fusion techniques. This unified training allows for a deeper understanding of the relationships between different modalities. Furthermore, Llama 4 benefits from an improved vision encoder, enhancing its visual understanding capabilities.

A novel training technique called MetaP was developed to reliably set crucial model hyperparameters, ensuring consistent performance across different model sizes and training scales. Efficiency was a key focus, with FP8 precision utilized during the training of Llama 4 Behemoth, achieving impressive FLOPs utilization.

Post-training details

Meta AI has also revamped its post-training pipeline to fine-tune Llama 4 for optimal performance in various applications. This pipeline follows a strategic curriculum, moving from lightweight SFT to online RL with a focus on harder prompts, and finally to lightweight DPO.

To address over-constraining of the model due SFT and DPO, they curated the dataset by filtering out more than 50% of their data, deemed "easy" by Llama judge models. The remaining harder data was used for lightweight DPO.

They selected harder prompts for the next online RL stage and got better a step up in performance. Furthermore they introduced continuous online RL, which alternated between training the model and then using it to continually filter and retain only medium-to-hard difficulty prompts.

The Power of MoE: Efficiency and Expertise

The introduction of the Mixture-of-Experts architecture in Llama 4 is a game-changer. In MoE models, only a fraction of the total parameters are active for any given input token. This selective activation leads to several benefits:

• Efficiency: By activating only a subset of experts, Llama 4 can process data more efficiently, reducing computational costs and memory requirements.
• Specialization: Different experts can specialize in different tasks or modalities, allowing Llama 4 to excel in a wide range of applications.
• Scalability: The MoE architecture allows for easy scaling of the model by adding more experts without significantly increasing the overall model size.

Impressive Results and Open Availability

The initial results for Llama 4 are compelling. Llama 4 Scout is supposedly the best multimodal model in its class, outperforming previous Llama generations and rivals like Gemma 3 and Gemini 2.0 Flash-Lite on various benchmarks. Llama 4 Maverick also stands out as a top performer, even surpassing GPT-4o and Gemini 2.0 Flash in many areas while achieving comparable results to DeepSeek v3 on reasoning and coding with significantly fewer active parameters.

Llama 4 Benchmarks (Source: Meta AI Blog)

Use Cases and Applications

Llama 4 Scout boasts a significant 256K context length from pre-training through post-training, unlocking remarkable ability to handle and understand long sequences of information. This advanced length generalization is evident in its strong performance on tasks requiring information retrieval within vast textual data ("needle in haystack") and its capability to process and analyze massive code repositories extending up to 10 million tokens.

Use and Availability

Llama 4 models are available on llama.com and HuggingFace. You can try chatting with the models on meta.ai.

For hosted Llama 4 API endpoints, you can go to Groq.

Credits

This blog is based on the Llama 4 announcement.