OLMoE 1B-7B

OLMoE 1B-7B needs an estimated 5.3 GB at Q4, 9 GB at Q8 and 16.4 GB at FP16. These estimates cover the 6.9B parameters as weights plus about 15% and 0.5 GB for a short context.

Yes, OLMoE 1B-7B fits on all four. A 24 GB GPU fits it at FP16 (estimated 16.4 GB), a 16 GB or 12 GB GPU fits it at Q8 (estimated 9 GB) and an 8 GB GPU fits it at Q4 (estimated 5.3 GB).

Yes, OLMoE 1B-7B can run on a Mac with unified memory. Assuming macOS lets the GPU use about 75% of unified memory, the smallest common Mac memory size that fits Q4 (estimated 5.3 GB) is 8 GB.

Yes, you can self-host OLMoE 1B-7B for full control of your data, since it has open weights.

Run OLMoE 1B-7B locally

No, OLMoE 1B-7B only accepts text input. It cannot read images, audio, video or PDFs.

The closest alternatives, matched on model kind and release date with no comparable price, are Pixtral 12B by Mistral AI, Nemotron Mini 4B Instruct by NVIDIA and Qwen2.5 7B Instruct by Alibaba.

Pixtral 12BOLMoE 1B-7B vs Pixtral 12BNemotron Mini 4B InstructOLMoE 1B-7B vs Nemotron Mini 4B InstructQwen2.5 7B InstructOLMoE 1B-7B vs Qwen2.5 7B Instruct

OLMoE 1B-7B has open weights, not fully open source. It is released under the Apache 2.0 license.

Yes, OLMoE 1B-7B uses the Apache 2.0 license, a permissive license that allows commercial use.

Download the OLMoE 1B-7B weights from the Hugging Face repository allenai/OLMoE-1B-7B-0924-Instruct.

Run OLMoE 1B-7B locally

OLMoE 1B-7B was made by Ai2 and released on September 4, 2024.

More from Ai2

Yes, OLMoE 1B-7B is free to use, as it has open weights.