Aleph Alpha releases Kolibri, a 78B MoE model with 1M context window
Aleph Alpha has released Kolibri, an English and German mixture-of-experts model with 78.1 billion parameters, 3.46 billion active per token, open weights under Apache 2.0 and up to 1M tokens of context. It was trained to 262,144 tokens and reaches a million by extrapolation. We cover the architecture, the long-context scores, benchmarks against Qwen, Nemotron and Mistral, hardware needs, the licence and the Cohere merger.
