Self-Hosted Inference Platform
LOCAL AI INFRASTRUCTURE
Self-hosted local inference platform using llama.cpp, NVIDIA GPU acceleration, OpenAI-compatible APIs, and Qwen models.
- Status
- ACTIVE
- Host
- UBUNTU
- Deployment
- SELF-HOSTED
Overview
System Overview
A self-hosted local inference platform using llama.cpp, NVIDIA GPU acceleration, OpenAI-compatible APIs, and Qwen models — the AI infrastructure layer that other projects, including Graydragon, build on.
GPU
NVIDIA GTX 1080 8GB
MODEL
Qwen3-8B Q3_K_L
INFERENCE
llama.cpp CUDA
API
OpenAI-compatible
CONTEXT
4096 tokens
Problem
The Problem
- ›Depending entirely on hosted AI providers introduces cost, privacy, and availability tradeoffs for daily-use automation.
- ›Consumer-grade GPU hardware (an 8GB GTX 1080) constrains which models and quantizations are practical to run locally.
- ›Applications built on top of AI shouldn't need to know or care whether inference is local or hosted.
Requirements
What It Had to Do
- ›Run a capable open-weight model locally within an 8GB GPU's memory budget.
- ›Expose inference through an OpenAI-compatible API so existing tooling and libraries work without modification.
- ›Keep the provider layer swappable so local and hosted inference can be used interchangeably.
Architecture
How It's Structured
Build
Engineering the Build
llama.cpp with CUDA acceleration serves a Qwen3-8B model at Q3_K_L quantization on an NVIDIA GTX 1080, chosen to fit within 8GB of VRAM while remaining usable for real tasks.
The server exposes an OpenAI-compatible API, so anything built to talk to a hosted provider can point at this endpoint instead with no code changes.
A 4096-token context window is treated as a known constraint that shapes how requests are structured upstream, rather than something to work around silently.
Challenges
What Made It Hard
- ›Fitting a useful model and context window into an 8GB consumer GPU without an unacceptable quality tradeoff.
- ›Keeping the OpenAI-compatible surface close enough to the real API that downstream tools don't need special cases.
- ›Deciding what should run locally versus fall back to a hosted provider.
Current System
Results — Where It Stands Today
- ›The GTX 1080 serves Qwen3-8B through llama.cpp behind an OpenAI-compatible endpoint in daily use.
- ›Graydragon and other internal tooling consume this endpoint as a first-class provider, not an experiment.
What's Next
What's Next
- ›Evaluate additional quantization and model options as they become practical on the existing hardware.
- ›Expand context-window handling for longer-running tasks.
- ›Continue reducing reliance on hosted providers where local inference is sufficient.