Can I fine-tune an LLM on an AMD GPU?

Yes, on Linux with ROCm. The same training code that runs on NVIDIA CUDA runs on AMD ROCm - PyTorch abstracts the backend. Quantinia is tested on the Radeon RX 7900 XTX.

Do I need to write code to fine-tune an LLM?

No. Libraries like Unsloth or the Hugging Face stack require Python. Quantinia is a desktop app: you install it, drop in your examples, and it produces a fine-tuned model - no notebook, no environment setup.

Guide · updated 2026-07-23

How to fine-tune an LLM on your own GPU (NVIDIA or AMD)

Q: How much VRAM do I need to fine-tune an LLM?

As a rough guide with LoRA in bf16: 8 GB handles models up to ~1.5B, 12-16 GB up to ~3B, and 24 GB up to ~7-8B. With 4-bit quantization (QLoRA) the ceiling roughly quadruples - a 24 GB card reaches ~30B-class models. LoRA and quantization are what make local training realistic.

Q: Can I fine-tune a large, agent-grade model on my own GPU?

Yes. With 4-bit quantization, a single 24 GB card can fine-tune ~30B-class models - large enough to drive an agent. Quantinia fine-tunes a 27B Qwen model on a single Radeon RX 7900 XTX, a consumer AMD card.

Ask on any forum whether you can fine-tune a language model on your own machine and the top reply is usually "your PC isn't powerful enough - use the cloud." That advice is out of date. With the right technique, a single consumer GPU is enough to fine-tune real models - and you keep your data and your model on hardware you own.

Can your GPU actually do it?

The reason people think local fine-tuning is impossible is that they picture full fine-tuning - updating every weight in the model. That does need a datacenter. But you almost never need it. LoRA (Low-Rank Adaptation) trains a tiny set of extra weights on top of a frozen base model. It reaches most of the quality of full fine-tuning while using a fraction of the memory.

A realistic VRAM guide with LoRA in bf16:

8 GB - models up to ~1.5B parameters (great for testing and small tasks)
12-16 GB - up to ~3B parameters
24 GB (e.g. RTX 4090, RX 7900 XTX) - up to ~7-8B parameters

If you have a gaming GPU from the last few years, you can fine-tune something useful today.

Go bigger: agent-grade models, not toys

Here is where it gets interesting. Those numbers are for standard bf16. Drop to 4-bit quantization (QLoRA) and the ceiling jumps: you quantize the frozen base model to 4-bit, so a model that would need ~48 GB in bf16 fits in roughly a quarter of that. The LoRA adapters you train stay full precision, so quality holds up.

That changes the whole conversation. A single 24 GB card no longer tops out at 8B - it reaches ~30B-class models, the size where a model is genuinely capable of driving an agent: tool use, multi-step reasoning, structured output. You are no longer limited to fine-tuning toys on your own hardware; you can shape a real, agent-grade model on it.

This is not theory. We fine-tune a 27B Qwen model on a single Radeon RX 7900 XTX - one 24 GB consumer AMD card - and it runs well. A capable, agentic-scale model, trained on hardware you can buy off the shelf, with your data never leaving the machine.

What you actually give it

Fine-tuning is teaching a model your examples. You provide input/output pairs - a question and the answer you want, an instruction and the ideal response, a chat and how the assistant should reply. A few dozen good examples already move a model; a few hundred to a few thousand is a strong dataset. The model learns your style, your format, your domain.

NVIDIA (CUDA) - Windows or Linux

If you have an NVIDIA card, you are on the most common path. Training runs through CUDA, and it works on both Windows and Linux. The building blocks are PyTorch (the CUDA build), plus the Hugging Face libraries (transformers, PEFT, accelerate). The model, your data, and the training all stay on your machine.

This is the mainstream route - the overwhelming majority of local fine-tuning happens on NVIDIA GPUs, from a laptop 3060 to an RTX 4090.

AMD (ROCm) - Linux

Here is the part most guides skip: you do not need NVIDIA. On Linux, AMD GPUs train through ROCm, AMD's equivalent of CUDA. The key insight is that PyTorch abstracts the backend - the same training code runs on CUDA and ROCm, you just install the ROCm build of PyTorch. Quantinia is tested end-to-end on the Radeon RX 7900 XTX, and the identical pipeline runs on NVIDIA. One tool, either card.

(On Windows, AMD training support is still thin, so AMD fine-tuning means Linux for now. NVIDIA covers both.)

The hard part nobody warns you about

The training itself is a few lines. The pain is the environment: matching the right PyTorch build to your exact GPU and driver, pinning versions of transformers and PEFT so they don't break each other, wrangling CUDA or ROCm, and debugging cryptic errors before you have trained a single step. This is where most people give up and retreat to a cloud notebook.

The no-code way

This is exactly the gap Quantinia closes. It is a desktop app, not a library or a notebook:

Install it on Windows or Linux - a wizard detects your GPU and sets up the right PyTorch build automatically.
Drop in your examples.
Get a fine-tuned model back. No terminal, no dependency hell, no Python.

To be straight about it: if you already live in Python and want maximum speed, an optimized library like Unsloth will train faster. Quantinia is not trying to win on raw throughput. It wins on ease, ownership, and running the same on any GPU - you own the model, your data never leaves your machine, and it just works whether your card is green or red.

Download Quantinia

FAQ

Can you fine-tune an LLM on your own PC?

Yes. With LoRA, a consumer GPU with 8-24 GB of VRAM can fine-tune models from 0.5B up to 8B parameters. You need the right technique and the right build of PyTorch for your GPU, not a datacenter.

How much VRAM do I need?

With LoRA in bf16, roughly: 8 GB up to ~1.5B, 12-16 GB up to ~3B, 24 GB up to ~7-8B. With 4-bit quantization (QLoRA) the ceiling roughly quadruples - a 24 GB card reaches ~30B-class models. Full fine-tuning needs far more; LoRA and quantization are what make local training realistic.

Can I fine-tune a large, agent-grade model on my own GPU?

Yes. Using 4-bit quantization, a single 24 GB card can fine-tune ~30B-class models - large enough to drive an agent (tool use, multi-step reasoning). We fine-tune a 27B Qwen model on a single Radeon RX 7900 XTX, a consumer AMD card.

Can I fine-tune on an AMD GPU?

Yes, on Linux with ROCm. The same code that runs on NVIDIA CUDA runs on AMD ROCm. Quantinia is tested on the RX 7900 XTX.

Do I need to write code?

No. Unsloth and the Hugging Face stack need Python. Quantinia is a desktop app - install, add examples, get a model.