Self-Hosted Inference Platform

LOCAL AI INFRASTRUCTURE

Self-hosted local inference platform using llama.cpp, NVIDIA GPU acceleration, OpenAI-compatible APIs, and Qwen models.

LLMCUDALLAMA.CPPSELF-HOSTEDAI

Status: ACTIVE
Host: UBUNTU
Deployment: SELF-HOSTED

Local AI Gateway: graydragon-ai node running Qwen3-8B Q3_K_L on a GTX 1080 8GB, serving local, private, GPU-accelerated inference.

Overview

System Overview

A self-hosted local inference platform using llama.cpp, NVIDIA GPU acceleration, OpenAI-compatible APIs, and Qwen models — the AI infrastructure layer that other projects, including Graydragon, build on.

GPU

NVIDIA GTX 1080 8GB

MODEL

Qwen3-8B Q3_K_L

INFERENCE

llama.cpp CUDA

API

OpenAI-compatible

CONTEXT

4096 tokens

llama.cppCUDANVIDIA GTX 1080 8GBQwen3-8BOpenAI-compatible API

Problem

The Problem

›Depending entirely on hosted AI providers introduces cost, privacy, and availability tradeoffs for daily-use automation.
›Consumer-grade GPU hardware (an 8GB GTX 1080) constrains which models and quantizations are practical to run locally.
›Applications built on top of AI shouldn't need to know or care whether inference is local or hosted.

Requirements

What It Had to Do

›Run a capable open-weight model locally within an 8GB GPU's memory budget.
›Expose inference through an OpenAI-compatible API so existing tooling and libraries work without modification.
›Keep the provider layer swappable so local and hosted inference can be used interchangeably.

Architecture

How It's Structured

Qwen3-8B Model

llama.cpp (CUDA)

NVIDIA GTX 1080

OpenAI-Compatible API

Downstream Applications

Build

Engineering the Build

llama.cpp with CUDA acceleration serves a Qwen3-8B model at Q3_K_L quantization on an NVIDIA GTX 1080, chosen to fit within 8GB of VRAM while remaining usable for real tasks.

The server exposes an OpenAI-compatible API, so anything built to talk to a hosted provider can point at this endpoint instead with no code changes.

A 4096-token context window is treated as a known constraint that shapes how requests are structured upstream, rather than something to work around silently.

Challenges

What Made It Hard

›Fitting a useful model and context window into an 8GB consumer GPU without an unacceptable quality tradeoff.
›Keeping the OpenAI-compatible surface close enough to the real API that downstream tools don't need special cases.
›Deciding what should run locally versus fall back to a hosted provider.

Current System

Results — Where It Stands Today

›The GTX 1080 serves Qwen3-8B through llama.cpp behind an OpenAI-compatible endpoint in daily use.
›Graydragon and other internal tooling consume this endpoint as a first-class provider, not an experiment.

What's Next

›Evaluate additional quantization and model options as they become practical on the existing hardware.
›Expand context-window handling for longer-running tasks.
›Continue reducing reliance on hosted providers where local inference is sufficient.

Back to Projects