Skip to content

Matt Adam

Tech Blog

Menu
  • Home
  • Home Lab
    • Home Lab – Gen 1
    • Home Lab – Gen 2
    • Home Lab – Gen 3
  • VMware
    • VMware Cloud Foundation (VCF)
    • Avi Networks
  • Kubernetes
    • Openshift
    • K3s
    • Tanzu
      • Tanzu – TKGs (WCP)
      • Tanzu – TKG (TKGm)
    • Avi Kubernetes Operator (AKO)
  • About
  • Privacy Policy
Menu

How to Run a Local LLM in Your Home Lab with Ollama and Open WebUI

Posted on November 17, 2026November 12, 2026 by Matt Adam

I’ve been running AI chat tools for a while now, and the one thing that kept bugging me was where all my prompts were going. Config files, log snippets, half-finished scripts with internal hostnames in them. All of it shipped off to someone else’s cloud. If that sounds familiar, the good news is you can run a local LLM in your home lab today with nothing more than a decent box, Docker, and about 30 minutes.

In this guide I’ll walk you through setting up Ollama (the engine that actually runs the models) and Open WebUI (a ChatGPT-style web interface) on a home lab server. By the end you’ll have your own private AI assistant that runs on your hardware, works offline, and costs nothing per prompt.

Table of Contents

Toggle
  • Why Run a Local LLM in Your Home Lab?
  • Lab Environment and Software Versions
  • Hardware Requirements for Running LLMs Locally
  • Step 1: Install the NVIDIA Container Toolkit (GPU Users Only)
  • Step 2: Deploy Ollama and Open WebUI with Docker Compose
  • Step 3: Pull Your First Model
  • Best Local LLM Models for a Home Lab
  • Step 4: Set Up Open WebUI
  • Step 5: Access Your Private AI Securely
  • Troubleshooting Common Ollama Issues
  • Wrapping Up

Why Run a Local LLM in Your Home Lab?

A few reasons I hear over and over from people building these setups:

  • Privacy. Your prompts, documents, and chat history never leave your network. That’s a big deal if you’re pasting in work configs or anything with customer data.
  • No subscription fees. Once the hardware is paid for, every prompt is free. No API keys, no rate limits, no surprise bills.
  • Offline access. Internet goes down? Your AI assistant keeps working.
  • Learning. Running models yourself teaches you a ton about how LLMs actually work: quantization, context windows, VRAM limits, all of it. That’s real, resume-worthy experience.

Just to reiterate, a local model isn’t going to beat the biggest frontier models at everything. But for coding help, summarizing docs, writing scripts, and general Q&A, a well-chosen 8B to 14B model is surprisingly capable.

Lab Environment and Software Versions

Here’s what I’m using for this walkthrough. You don’t need to match it exactly, but it helps to know what was tested.

Component Version / Spec
Host OS Ubuntu Server 24.04 LTS
CPU AMD Ryzen 7 5700G (8 cores)
RAM 64 GB DDR4
GPU NVIDIA RTX 3060 12 GB
Docker Engine 27.x with Compose v2
NVIDIA driver 570 series
Ollama latest (ollama/ollama image)
Open WebUI main (ghcr.io/open-webui/open-webui)

If you don’t have Docker set up yet, start with my guide on setting up Docker in a home lab and come back here.

Hardware Requirements for Running LLMs Locally

This is the part most people get wrong, so let’s spend a minute on it. The single biggest factor in how well a local LLM runs is VRAM (the memory on your GPU). The model has to fit in memory to run fast. If it spills over into system RAM, it still works, but it gets slow.

Here’s a rough sizing guide using the common Q4 quantization that Ollama pulls by default:

Model size Approx. memory needed Good hardware match
1B to 3B 2 to 4 GB CPU only, mini PCs, Raspberry Pi 5
7B to 8B 5 to 6 GB 8 GB GPU (RTX 3060 Ti, 4060)
12B to 14B 8 to 10 GB 12 GB GPU (RTX 3060 12 GB, 4070)
27B to 32B 18 to 22 GB 24 GB GPU (RTX 3090, 4090)
70B 40+ GB Multi-GPU or a big unified memory Mac

No GPU? You can still run small models on CPU only. Expect a few words per second on a modern 8-core chip with a 3B model. Usable for testing, not great for daily use. If you’re shopping for hardware, my affordable home server build guide is a good starting point, just budget extra for a GPU with at least 12 GB of VRAM.

Step 1: Install the NVIDIA Container Toolkit (GPU Users Only)

If you’re running on CPU only, skip ahead to Step 2.

Docker containers can’t see your GPU by default. The NVIDIA Container Toolkit fixes that. First, make sure your driver is working on the host:

# Confirm the NVIDIA driver is loaded and the GPU is visible
nvidia-smi

You should see your card listed with the driver version. Now install the toolkit:

# Add NVIDIA's package repository and signing key
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | \
  sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install the toolkit
sudo apt update && sudo apt install -y nvidia-container-toolkit

# Tell Docker to use the NVIDIA runtime, then restart Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Quick sanity check that Docker can see the GPU:

# Should print the same nvidia-smi table, but from inside a container
docker run --rm --gpus all ubuntu nvidia-smi

If that works, you’re good to go.

Step 2: Deploy Ollama and Open WebUI with Docker Compose

I like running both pieces in one Compose file. It keeps everything together, makes upgrades a one-liner, and the two containers can talk to each other by name.

Create a folder for the stack:

mkdir -p ~/stacks/local-ai && cd ~/stacks/local-ai

Then create a docker-compose.yml file:

services:
  ollama:
    image: ollama/ollama:latest          # The LLM engine
    container_name: ollama
    restart: unless-stopped
    ports:
      - "11434:11434"                    # Ollama API (handy for other apps and scripts)
    volumes:
      - ollama_data:/root/.ollama        # Downloaded models live here, keep them persistent
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia             # Remove this whole deploy block for CPU-only hosts
              count: all
              capabilities: [gpu]

  open-webui:
    image: ghcr.io/open-webui/open-webui:main   # ChatGPT-style web interface
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"                      # Browse to http://<server>:3000
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434   # Points the UI at the Ollama container by name
    volumes:
      - open_webui_data:/app/backend/data     # Users, chats, and settings
    depends_on:
      - ollama

volumes:
  ollama_data:
  open_webui_data:

Bring it up:

docker compose up -d

Give it a minute on the first run. Open WebUI is a fairly large image.

Pro tip: Once you’ve got a setup you like, pin the image tags to specific versions instead of latest and main. That way a surprise update doesn’t break things on a Saturday morning.

Step 3: Pull Your First Model

Ollama doesn’t ship with any models, so let’s grab one. You can do this from the command line:

# Pull a solid general-purpose 8B model
docker exec -it ollama ollama pull qwen3:8b

# List what's downloaded
docker exec -it ollama ollama list

Then confirm it’s actually running on the GPU:

# Run a quick prompt, then check where the model is loaded
docker exec -it ollama ollama run qwen3:8b "Explain VLANs in two sentences."
docker exec -it ollama ollama ps

In the ollama ps output, look at the PROCESSOR column. You want to see 100% GPU. If it says CPU, go back and double-check the NVIDIA toolkit step.

Best Local LLM Models for a Home Lab

The model library changes fast, so treat this as a starting point. These are the ones I’d try first, grouped by what they’re good at:

Model Size Best for
llama3.2:3b ~2 GB Low-power boxes, quick tests, CPU-only setups
qwen3:8b ~5 GB Great all-rounder, strong at reasoning and code
gemma3:12b ~8 GB Writing, summarizing, and it handles images too
qwen2.5-coder:14b ~9 GB Coding help, scripting, Ansible and Terraform snippets
gemma3:27b ~17 GB Best quality if you’ve got a 24 GB card
nomic-embed-text small Embeddings for document search (RAG)

My advice: start with an 8B model, get a feel for speed and quality, then step up as your hardware allows. Bigger isn’t always better if it means waiting 30 seconds for every answer.

Step 4: Set Up Open WebUI

Open a browser and go to http://192.168.1.50:3000 (swap in your server’s IP).

The first account you create becomes the admin, so do that right away before anyone else on your network finds the page. After you’re logged in:

  1. Click the model dropdown at the top and pick qwen3:8b.
  2. Ask it something. You should get a response streaming back within a second or two.
  3. Head into Admin Panel > Settings > Models to pull more models straight from the UI without touching the terminal.

A couple of features worth turning on early:

  • Document upload (RAG). Drop in a PDF or text file and ask questions about it. Great for vendor docs and runbooks.
  • Multiple users. Give family members or teammates their own accounts so their chat history stays separate.
  • Model presets. Create a custom “Home Lab Helper” model with a system prompt like “You are a Linux and networking expert. Keep answers short and include commands.”

Step 5: Access Your Private AI Securely

Right now Open WebUI is only reachable on your LAN, and honestly that’s the safest place for it. Please don’t just port-forward 3000 on your router.

If you want access while you’re away from home, you’ve got two good options:

  • VPN in. This is my preference. Check out my guide on self-hosting a VPN in your home lab.
  • Reverse proxy with HTTPS. Put it behind a reverse proxy with a proper SSL certificate and strong authentication. I’m covering exactly how to do that in next week’s post on Nginx Proxy Manager.

Troubleshooting Common Ollama Issues

A few things I’ve run into, so you don’t have to:

  • Model is painfully slow. It’s probably running on CPU or spilling out of VRAM. Run ollama ps and check the PROCESSOR column. Drop to a smaller model or a smaller quantization.
  • Open WebUI shows no models. The UI can’t reach Ollama. Make sure OLLAMA_BASE_URL uses the container name (http://ollama:11434), not localhost.
  • Out of disk space. Models add up fast. Clean house with docker exec -it ollama ollama rm <model>.
  • GPU not detected after a driver update. Reboot the host, then restart the stack with docker compose down && docker compose up -d.

Wrapping Up

That’s it. You’ve now got a private, self-hosted AI assistant running in your home lab, with a clean web UI and an API on port 11434 that your other scripts and tools can hit. No subscriptions, no data leaving your network.

From here, try pointing an editor plugin or a Python script at the Ollama API, or load up your lab documentation and let it answer questions for you. Feel free to customize the Compose file and model lineup for your own hardware and use case.

Leave a Reply Cancel reply

You must be logged in to post a comment.

Social Media

  • LinkedIn
  • X
  • Bluesky
  • Mastodon

Recent Posts

  • How to Run a Local LLM in Your Home Lab with Ollama and Open WebUI
  • I Built a Strategy Game That Ends When the Ice Wins. I Need People to Break It.
  • Building a Boat? Someone After You Will Wish You’d Written This Down.
  • Financial Solutions for ETF share classes
  • Power Consumption Tips for Efficient Home Labs: Save Energy, Save Money
© 2026 Matt Adam | Powered by Minimalist Blog WordPress Theme