Llama 3
Meta's Llama 3 is the latest iteration in its series of large language models, boasting significant advancements in AI capabilities.
The first version of the Llama models was released in February of last year as one of the first open weight large language models. Subsequently, it launched the second version in July 2023.
Open weight models are fundamentally different from open source models, which include training source code, weights that can be compared to software executables or binaries, and inference code that allows developers to use the model.
With Llama, Meta has not released the source code or the dataset used to train the models. It only published the weights and the inference code in the public domain. But, that in itself is a significant step in enabling ML researchers to extend the models to create new variants of them.
Llama 2 has gained widespread acceptance among researchers and developers. Dozens of fine-tuned variants have been created and used in production.
With the latest iteration, Meta has pushed the envelope to make Llama powerful. Compared to other open models such as Mistral and Gemma, Llama 3 scores better in the majority of the performance benchmarks.
Llama 3 is becoming the foundation of Meta's new AI assistant, demonstrating the model's reliability and accuracy.
Here are 10 essential facts about Llama 3:
1. It introduces four new models based on the Llama 2 architecture, available in two sizes: 8 billion (8B) and 70 billion (70B) parameters. Each size offers a base model and an instruction-tuned version, designed to enhance performance in specific tasks. The instruction-tuned version is meant for powering chatbots that can have a conversation with users. The number of parameters is directly proportional to the size of the training dataset, which means the 70B model performs better than its smaller counterpart.
2. Llama 3 powers Meta AI, the company's brand-new assistant. The chatbot is available on Meta AI on Facebook, Instagram, WhatsApp and Messenger. It is also embedded in the search experience across Facebook, Instagram, WhatsApp and Messenger.
3. All variants of Llama 3 support a context length of 8,000 tokens, allowing for more extended interactions and more complex input handling compared to many previous models. More tokens mean more content that includes both the input prompt from the users and the response from the model. Token roughly translates to a word or a subset of a word. With 8000 tokens, users can send larger prompts and expect the model to generate more content in response to them. In comparison, the previous version of Llama supported only 4096 tokens.
4. Llama 3 models are integrated into the Hugging Face ecosystem, making them readily available to developers. Hugging Face has become the defacto platform for open model providers such as Meta and Msitral to publish their models and datasets. Developers and researchers rely on Hugging Face to download these models. This integration with Hugging Face includes tools like transformers and inference endpoints, facilitating easier adoption and application development. Llama 3 is also available from model-as-a-service providers such as Perplexity Labs and Fireworks.ai, as well as cloud provider platforms such as Azure ML and Vertex AI.
5. Alongside the Llama 3 models, Meta has released Llama Guard 2, a safety model fine-tuned on the 8B version, designed to improve the production use cases' safety and reliability. This applies the required guardrails to ensure that the model adheres to the safety policies that are predefined.
6. The Llama 3 models have shown impressive performance across various benchmarks. The 70B model, for instance, outperforms other high-profile models like OpenAI's GPT-3.5 and Google's Gemini on tasks including coding, creative writing and summarization.
7. The models were trained on a dataset comprising 15 trillion tokens, which is about seven times the size of the dataset used for Llama 2. This extensive training has significantly contributed to the models' improved performance and capabilities. They are trained on purpose-built GPU clusters recently built by Meta.
8. Meta is actively developing more capable versions of Llama 3, with future models expected to exceed 400 billion parameters. These versions aim to support multiple languages and modalities, enhancing the model's versatility and applicability across different regions and formats. The larger model variant is expected to become available later this year.
9. Meta continues to emphasize its commitment to the open-source community by making Llama 3 available for free. This approach not only fosters innovation but also allows for widespread testing and improvement by developers worldwide. Interestingly, Meta calls Llama 3 an openly accessible model without calling it an open source model.
10. Llama 3 models are optimized for hardware from Intel, AMD and Nvidia. Intel has published a detailed guide on the performance of the model on its Gaudi AI accelerators and Xeon CPUs.
Llama 3 marks a significant step forward in the evolution of open models. Given Meta's reach and deep partnerships with major industry players, the model is expected to gain widespread adoption in the coming months.
Update at 1:20 AM EST: This article was updated to add additional details and explanation.
