jhu-clsp

Encoders experience a renaissance since the introduction of ModernBERT, which is one of the most downloaded models every month on HuggingFace. mmBERT is a multilingual version, supports 8K context and over 1800 languages due to its training on 3T tokens.

An open source alternative (yes, with data!) for ModernBERT. They also include 250+ checkpoints, so anyone can use the models from different points across the training phases.