An ultra-sparse MoE with 400B total and 13B active parameters, trained by an American company. They also released a tech report and two base models, one "true" base model pre-annealing and the base model after the pre-training phase. Many more insights, including technical details and their motivation, can be found in our interview with the founders and pre-training lead:
Specs
Params398B total, 13B active
LicenseApache-2.0
Tags
modelsmodels/llmsmultilingualartifacts/18curated/our-picks
Related Models
