MLX
Apple machine-learning framework used for efficient inference on Apple silicon.
macOS
Deployment
Runtimes
The software layer that determines how models actually run locally, on workstations, and in production serving environments.
Ollama
Local model runner focused on straightforward installation and model management.
macOSLinuxWindows
TensorRT-LLM
NVIDIA inference stack for optimized large-model serving on supported GPUs.
Linux
llama.cpp
Portable local inference runtime for GGUF models across CPU and accelerator backends.
macOSLinuxWindows
vLLM
High-throughput inference and serving runtime for production model deployments.
Linux