I’ve been running AI chat tools for a while now, and the one thing that kept bugging me was where all my prompts were going. Config files, log snippets, half-finished scripts with internal hostnames in them. All of it shipped off to someone else’s cloud. If that sounds familiar, the good news is you can run a local LLM in your home lab today with nothing more than a decent box, Docker, and about 30 minutes.
In this guide I’ll walk you through setting up Ollama (the engine that actually runs the models) and Open WebUI (a ChatGPT-style web interface) on a home lab server. By the end you’ll have your own private AI assistant that runs on your hardware, works offline, and costs nothing per prompt.
Why Run a Local LLM in Your Home Lab?
A few reasons I hear over and over from people building these setups:
- Privacy. Your prompts, documents, and chat history never leave your network. That’s a big deal if you’re pasting in work configs or anything with customer data.
- No subscription fees. Once the hardware is paid for, every prompt is free. No API keys, no rate limits, no surprise bills.
- Offline access. Internet goes down? Your AI assistant keeps working.
- Learning. Running models yourself teaches you a ton about how LLMs actually work: quantization, context windows, VRAM limits, all of it. That’s real, resume-worthy experience.
Just to reiterate, a local model isn’t going to beat the biggest frontier models at everything. But for coding help, summarizing docs, writing scripts, and general Q&A, a well-chosen 8B to 14B model is surprisingly capable.
Lab Environment and Software Versions
Here’s what I’m using for this walkthrough. You don’t need to match it exactly, but it helps to know what was tested.
| Component | Version / Spec |
|---|---|
| Host OS | Ubuntu Server 24.04 LTS |
| CPU | AMD Ryzen 7 5700G (8 cores) |
| RAM | 64 GB DDR4 |
| GPU | NVIDIA RTX 3060 12 GB |
| Docker Engine | 27.x with Compose v2 |
| NVIDIA driver | 570 series |
| Ollama | latest (ollama/ollama image) |
| Open WebUI | main (ghcr.io/open-webui/open-webui) |
If you don’t have Docker set up yet, start with my guide on setting up Docker in a home lab and come back here.
Hardware Requirements for Running LLMs Locally
This is the part most people get wrong, so let’s spend a minute on it. The single biggest factor in how well a local LLM runs is VRAM (the memory on your GPU). The model has to fit in memory to run fast. If it spills over into system RAM, it still works, but it gets slow.
Here’s a rough sizing guide using the common Q4 quantization that Ollama pulls by default:
| Model size | Approx. memory needed | Good hardware match |
|---|---|---|
| 1B to 3B | 2 to 4 GB | CPU only, mini PCs, Raspberry Pi 5 |
| 7B to 8B | 5 to 6 GB | 8 GB GPU (RTX 3060 Ti, 4060) |
| 12B to 14B | 8 to 10 GB | 12 GB GPU (RTX 3060 12 GB, 4070) |
| 27B to 32B | 18 to 22 GB | 24 GB GPU (RTX 3090, 4090) |
| 70B | 40+ GB | Multi-GPU or a big unified memory Mac |
No GPU? You can still run small models on CPU only. Expect a few words per second on a modern 8-core chip with a 3B model. Usable for testing, not great for daily use. If you’re shopping for hardware, my affordable home server build guide is a good starting point, just budget extra for a GPU with at least 12 GB of VRAM.
Step 1: Install the NVIDIA Container Toolkit (GPU Users Only)
If you’re running on CPU only, skip ahead to Step 2.
Docker containers can’t see your GPU by default. The NVIDIA Container Toolkit fixes that. First, make sure your driver is working on the host:
# Confirm the NVIDIA driver is loaded and the GPU is visible
nvidia-smi
You should see your card listed with the driver version. Now install the toolkit:
# Add NVIDIA's package repository and signing key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
# Install the toolkit
sudo apt update && sudo apt install -y nvidia-container-toolkit
# Tell Docker to use the NVIDIA runtime, then restart Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
Quick sanity check that Docker can see the GPU:
# Should print the same nvidia-smi table, but from inside a container
docker run --rm --gpus all ubuntu nvidia-smi
If that works, you’re good to go.
Step 2: Deploy Ollama and Open WebUI with Docker Compose
I like running both pieces in one Compose file. It keeps everything together, makes upgrades a one-liner, and the two containers can talk to each other by name.
Create a folder for the stack:
mkdir -p ~/stacks/local-ai && cd ~/stacks/local-ai
Then create a docker-compose.yml file:
services:
ollama:
image: ollama/ollama:latest # The LLM engine
container_name: ollama
restart: unless-stopped
ports:
- "11434:11434" # Ollama API (handy for other apps and scripts)
volumes:
- ollama_data:/root/.ollama # Downloaded models live here, keep them persistent
deploy:
resources:
reservations:
devices:
- driver: nvidia # Remove this whole deploy block for CPU-only hosts
count: all
capabilities: [gpu]
open-webui:
image: ghcr.io/open-webui/open-webui:main # ChatGPT-style web interface
container_name: open-webui
restart: unless-stopped
ports:
- "3000:8080" # Browse to http://<server>:3000
environment:
- OLLAMA_BASE_URL=http://ollama:11434 # Points the UI at the Ollama container by name
volumes:
- open_webui_data:/app/backend/data # Users, chats, and settings
depends_on:
- ollama
volumes:
ollama_data:
open_webui_data:
Bring it up:
docker compose up -d
Give it a minute on the first run. Open WebUI is a fairly large image.
Pro tip: Once you’ve got a setup you like, pin the image tags to specific versions instead of latest and main. That way a surprise update doesn’t break things on a Saturday morning.
Step 3: Pull Your First Model
Ollama doesn’t ship with any models, so let’s grab one. You can do this from the command line:
# Pull a solid general-purpose 8B model
docker exec -it ollama ollama pull qwen3:8b
# List what's downloaded
docker exec -it ollama ollama list
Then confirm it’s actually running on the GPU:
# Run a quick prompt, then check where the model is loaded
docker exec -it ollama ollama run qwen3:8b "Explain VLANs in two sentences."
docker exec -it ollama ollama ps
In the ollama ps output, look at the PROCESSOR column. You want to see 100% GPU. If it says CPU, go back and double-check the NVIDIA toolkit step.
Best Local LLM Models for a Home Lab
The model library changes fast, so treat this as a starting point. These are the ones I’d try first, grouped by what they’re good at:
| Model | Size | Best for |
|---|---|---|
llama3.2:3b |
~2 GB | Low-power boxes, quick tests, CPU-only setups |
qwen3:8b |
~5 GB | Great all-rounder, strong at reasoning and code |
gemma3:12b |
~8 GB | Writing, summarizing, and it handles images too |
qwen2.5-coder:14b |
~9 GB | Coding help, scripting, Ansible and Terraform snippets |
gemma3:27b |
~17 GB | Best quality if you’ve got a 24 GB card |
nomic-embed-text |
small | Embeddings for document search (RAG) |
My advice: start with an 8B model, get a feel for speed and quality, then step up as your hardware allows. Bigger isn’t always better if it means waiting 30 seconds for every answer.
Step 4: Set Up Open WebUI
Open a browser and go to http://192.168.1.50:3000 (swap in your server’s IP).
The first account you create becomes the admin, so do that right away before anyone else on your network finds the page. After you’re logged in:
- Click the model dropdown at the top and pick
qwen3:8b. - Ask it something. You should get a response streaming back within a second or two.
- Head into Admin Panel > Settings > Models to pull more models straight from the UI without touching the terminal.
A couple of features worth turning on early:
- Document upload (RAG). Drop in a PDF or text file and ask questions about it. Great for vendor docs and runbooks.
- Multiple users. Give family members or teammates their own accounts so their chat history stays separate.
- Model presets. Create a custom “Home Lab Helper” model with a system prompt like “You are a Linux and networking expert. Keep answers short and include commands.”
Step 5: Access Your Private AI Securely
Right now Open WebUI is only reachable on your LAN, and honestly that’s the safest place for it. Please don’t just port-forward 3000 on your router.
If you want access while you’re away from home, you’ve got two good options:
- VPN in. This is my preference. Check out my guide on self-hosting a VPN in your home lab.
- Reverse proxy with HTTPS. Put it behind a reverse proxy with a proper SSL certificate and strong authentication. I’m covering exactly how to do that in next week’s post on Nginx Proxy Manager.
Troubleshooting Common Ollama Issues
A few things I’ve run into, so you don’t have to:
- Model is painfully slow. It’s probably running on CPU or spilling out of VRAM. Run
ollama psand check the PROCESSOR column. Drop to a smaller model or a smaller quantization. - Open WebUI shows no models. The UI can’t reach Ollama. Make sure
OLLAMA_BASE_URLuses the container name (http://ollama:11434), notlocalhost. - Out of disk space. Models add up fast. Clean house with
docker exec -it ollama ollama rm <model>. - GPU not detected after a driver update. Reboot the host, then restart the stack with
docker compose down && docker compose up -d.
Wrapping Up
That’s it. You’ve now got a private, self-hosted AI assistant running in your home lab, with a clean web UI and an API on port 11434 that your other scripts and tools can hit. No subscriptions, no data leaving your network.
From here, try pointing an editor plugin or a Python script at the Ollama API, or load up your lab documentation and let it answer questions for you. Feel free to customize the Compose file and model lineup for your own hardware and use case.