ModernBERT-base

As a blast to the past, Answer.AI has released a model based on the BERT architecture, which is still widely used in the enterprise context. The model uses a lot of modern techniques, i.e., RoPE for long context, Flash Attention and training on a lot of tokens (2 Trillion) for BERT standards. The accompanying paper is information-dense and goes in-depth on the technical decisions made.