Last verified: April 2026 $0 API Costs

Dify GPU Hosting Guide 2026 — Run Local LLMs with Dify

Host Dify on a GPU server and connect it to Ollama or LocalAI to run Llama 3, Mistral, and other open-source models locally — with zero per-token API costs and complete data privacy.

Why Run Dify on a GPU Server?

Connecting Dify to a locally-hosted LLM via Ollama or LocalAI removes dependence on cloud AI providers entirely. Here is what you gain:

💰

No API Costs

Pay only for the GPU server — not per token. High-volume usage becomes dramatically cheaper.

🔒

Data Privacy

Prompts and responses never leave your infrastructure — essential for regulated industries.

🧩

Custom Models

Run fine-tuned or domain-specific models that are not available through any public API.

🚀

No Rate Limits

Burst as many requests as your GPU can handle — no throttling, no quota errors.

GPU Cloud Providers Compared

Prices are approximate on-demand rates as of early 2026. Reserved and spot instances are typically cheaper.

Provider GPU VRAM Price/hr Best For
Lambda Labs A10 24 GB $0.75/hr Development
Vast.ai RTX 4090 24 GB ~$0.35/hr Budget
RunPod A100 80 GB $1.99/hr Production
CoreWeave H100 80 GB $2.50/hr Enterprise
Hetzner GPU A100 80 GB 2.49 EUR/hr EU compliance
1

Install CUDA and NVIDIA Container Toolkit

Before installing Dify or Ollama, you need the NVIDIA CUDA drivers and the Container Toolkit so Docker containers can access the GPU.

Install CUDA Toolkit 12.3

Verify GPU and Configure Docker

After running nvidia-smi, you should see your GPU listed with its driver version and VRAM. If Docker can now use --gpus all, you are ready for the next step.

2

Install Ollama and Pull LLM Models

Ollama is the easiest way to serve open-source LLMs on your GPU. It automatically detects CUDA and uses the GPU for inference.

Install Ollama and Pull Models

Bind Ollama to All Network Interfaces

By default Ollama only listens on localhost. To make it reachable from Dify's Docker containers, you need to bind it to 0.0.0.0:

Configure docker-compose.override.yaml

Create or edit docker-compose.override.yaml in your Dify directory so containers can resolve host.docker.internal to the host machine on Linux:

Note: On macOS and Windows, host.docker.internal resolves automatically. On Linux, the extra_hosts entry above is required.

3

Connect Dify to Ollama

With Ollama running and reachable, add it as a model provider inside Dify:

  1. Open your Dify instance and click your avatar in the top-right corner.
  2. Go to Settings then Model Provider.
  3. Scroll down to find Ollama and click Add Model.
  4. Set the Base URL to http://host.docker.internal:11434.
  5. Enter the Model Name exactly as listed by ollama list (e.g. llama3.1:8b).
  6. Click Save — Dify will test the connection. A green checkmark confirms success.
  7. The model is now available in all your Dify apps and workflows.

Tip: Repeat step 5 for each model you pulled. You can add as many Ollama models as you like — each appears as a separate selectable model within Dify.

4

LocalAI — An OpenAI-Compatible Alternative

If you prefer an OpenAI-compatible API surface, LocalAI is an excellent alternative to Ollama. It exposes endpoints like /v1/chat/completions so you can use Dify's existing OpenAI integration without any extra configuration.

Run LocalAI with Docker (GPU)

Once running, configure Dify with Model Provider: OpenAI-API-compatible, set the base URL to http://host.docker.internal:8080/v1, and use any model name you have loaded in LocalAI. No API key is required for local deployments.

Model Recommendations by Use Case

Choose your model based on available VRAM and the quality-speed tradeoff your application needs.

Model VRAM Required Speed Best For
llama3.1:8b ~6 GB Fast General purpose, chat
mistral:7b ~5 GB Very fast Speed-critical apps
codellama:13b ~10 GB Medium Code generation
llama3.1:70b ~40 GB Slow High-quality outputs
mixtral:8x7b ~26 GB Medium Balanced quality/speed

VRAM Quick Reference

~6 GB
7B Models
e.g. Llama 3.1 8B, Mistral 7B
~10 GB
13B Models
e.g. CodeLlama 13B
~20 GB
34B Models
e.g. CodeLlama 34B
~40 GB
70B Models
e.g. Llama 3.1 70B

These are approximate requirements for full-precision (fp16) inference. Quantized models (Q4/Q5) can reduce VRAM usage by 30–50%, allowing larger models to run on smaller GPUs.

Related Guides

Common Mistakes and How to Avoid Them

1. Incorrect CUDA Version

Using a CUDA version that is incompatible with your GPU or the installed drivers can lead to runtime errors.

Fix: Ensure that the installed CUDA version matches the requirements of the LLM you are using. For instance, Llama 3 requires CUDA 12.3. Verify with:

nvidia-smi

2. Insufficient GPU Memory

Attempting to load large models on a GPU with inadequate VRAM results in "out of memory" errors.

Fix: Check the VRAM requirements of your model. For example, Llama 3 needs at least 16GB of VRAM. Upgrade your GPU or use a model with lower memory requirements.

3. Docker Configuration Issues

Improper Docker setup can prevent containers from accessing the GPU, leading to errors like "No GPU found."

Fix: Ensure that Docker is configured correctly to use the GPU. Check your Docker daemon configuration in /etc/docker/daemon.json for:

{
            "runtimes": {
                "nvidia": {
                    "path": "nvidia-container-runtime",
                    "runtimeArgs": []
                }
            }
        }

4. Ignoring Resource Limits

Not setting resource limits can lead to performance degradation and crashes when running multiple containers.

Fix: Use Docker's resource constraints in your docker-compose.yml:

services:
            api:
                deploy:
                    resources:
                        limits:
                            cpus: '1.0'
                            memory: 8G

5. Not Monitoring Resource Usage

Failing to monitor GPU and CPU usage can lead to unexpected downtimes or performance issues.

Fix: Use tools like nvidia-smi and htop to keep track of resource usage regularly.