Painless Docker - 2nd Edition

Docker Model Runner Overview

Docker Model Runner (DMR) is a specialized CLI plugin and runtime environment designed to simplify the lifecycle of AI models within the Docker ecosystem.

Essentially, it treats AI models - like Large Language Models (LLMs) or Image Generators - similarly to how Docker treats standard application containers. It allows developers to pull, manage, and serve these models locally without needing to manually configure complex Python environments or specialized hardware drivers.

Using DMR, developers can easily download pre-trained models from Docker Hub and access a range of features including:

• Unified Model Management: You can pull AI models from Docker Hub (or other registries) just like you pull images. DMR handles the storage and caching locally.
• "On-Demand" Loading: To save system resources, models only load into your RAM or GPU memory when you actually send a request, and they unload automatically when idle.
• Standardized Access: It creates a local server that provides OpenAI- and Ollama-compatible APIs. This means any app or tool designed to talk to ChatGPT can talk to your local Docker-hosted model instead.
• Cross-Backend Support: It acts as a bridge to various "Inference Engines" (the math brains that run the models).

At the time of writing, Docker Model Runner supports popular inference engines like:

• llama.cpp