Nemotron-H-56B-Base-8K

A hybrid transformer-mamba model trained on a whopping 20T tokens. The report goes into greater detail. The models perform on the same level as an attention-only variant, while being more than twice as fast. The long-context benchmarks also look competitive, making this architecture a serious contender to replace models which use sliding window attention or similar variants. The released models, however, support only 8K context.