Amazon Web Services, announced that Technology Innovation Institute, TII, trained its open-source Falcon 40B model on AWS. Falcon 40B is a 40-billion-parameter large language model, LLM available under the Apache 2.0 license that ranked #1 in Hugging Face’s Open LLM Leaderboard.
It was trained on 1 trillion tokens on Amazon SageMaker, a fully managed service for developing, training, tuning, and hosting machine learning models, including LLMs. The launch represents a significant milestone in the UAE’s commitment to fostering AI innovation and scientific contributions in line with the UAE National AI Strategy 2031.
Customers can now deploy Falcon 40B from Amazon SageMaker JumpStart, a machine learning, ML hub that offers pre-trained models, giving customers access to Falcon 40B’s state-of-the-art accuracy and industry- performance without having to build their own model from scratch.
Dr Ebtesam Almazrouei, Executive Director, Acting Chief AI Researcher of AI Cross centre Unit and Project Lead for LLM Projects at TII, highlighted Falcon 40B in a new blogpost, stating: “We proudly announce the official open-source release of Falcon-40B, the world’s top-ranking open-source language model.”
Wojciech Bajda, Managing Director, Public Sector Middle East, and Africa at AWS, said: “We are proud to have collaborated with the Technology Innovation Institute on the development of the Falcon LLM 40B model, which leverages Amazon SageMaker to train a model. The open-source release of Falcon-40B enables organisations to leverage its exceptional capabilities and advance AI-driven solutions to create new opportunities for progress and advancements.”
Falcon 40B is an exceptional open-source model with 40B parameters, specifically designed as a causal decoder-only model. It was trained on a vast dataset of 1,000B tokens, including RefinedWeb enhanced with curated corpora. The model is available under the Apache 2.0 license, ensuring its accessibility and usability. The architecture of Falcon-40B is optimised for inference, incorporating FlashAttention and multi-query techniques.
Click below to share this article