Fine-Tune an LLM Privately on a Rented GPU: A Practical Guide

You can fine-tune an 8B open-weights model on your own data with QLoRA on one rented 24 GB GPU, and a typical run costs under a dollar. The harder questions come first: whether fine-tuning is the right fix at all (for facts, retrieval usually wins), and how your data stays private on a machine someone else owns.

This guide covers both, then the VRAM you need by model size, the current tools, a working training script, a worked cost and how to serve the result. Everything was checked in September 2026; sources are at the end.

Fine-tune, RAG or better prompts

Fine-tuning changes how a model behaves. It’s a poor way to teach it facts. Ovadia et al. compared the two for knowledge injection and found that RAG “consistently outperforms” unsupervised fine-tuning, “both for existing knowledge encountered during training and entirely new knowledge”. Their summary: LLMs struggle to learn new facts through fine-tuning.

So walk down this tree before you rent anything:

Good reasons to fine-tune:

• Strict output format. Extracting fields into your schema on every call, without a page of instructions in every prompt.
• Style and tone. Support replies that sound like your team, or reports in a fixed structure.
• A narrow task done by a small model. A tuned 8B model can replace a large general model for one job, which matters when you serve it on cheap hardware.
• Shorter prompts. Behavior learned in the weights doesn’t need to be repeated in every request.

LoRA and QLoRA

Full fine-tuning updates every weight, so the GPU has to hold gradients and optimizer state for all of them on top of the model. LoRA freezes the base model and trains small low-rank matrices next to its layers; the original paper reported 10,000 times fewer trainable parameters and 3 times less GPU memory than full fine-tuning GPT-3 175B with Adam.

QLoRA goes further: the frozen base model is loaded in 4-bit NF4 precision, and only the adapters are trained in 16-bit. Dettmers et al. used it to fine-tune a 65B model on a single 48 GB GPU “while preserving full 16-bit finetuning task performance”. The paper added three pieces that the tools still use: the NF4 data type, double quantization of the quantization constants, and paged optimizers that absorb memory spikes.

The result of either is an adapter, a folder of a few tensors, that you apply on top of the unchanged base model. You can keep it separate or merge it into the weights. Image models use the same method: a Stable Diffusion LoRA trains on one rented 24 GB card for well under $10.

How much VRAM you need

Unsloth publishes a table of the minimum VRAM for fine-tuning by model size. These are its numbers, with its memory optimizations; plain Hugging Face training needs more, and longer sequences or larger batches push every row up.

Model size	QLoRA (4-bit)	LoRA (16-bit)	Rented card that fits QLoRA comfortably
3B	3.5 GB	8 GB	Any 12 GB+ card
8B	6 GB	22 GB	RTX 3090 / 4090 (24 GB)
14B	8.5 GB	33 GB	RTX 3090 / 4090 (24 GB)
32B	26 GB	76 GB	48 GB card (RTX A6000, A40, L40S)
70B	41 GB	164 GB	80 GB card (A100, H100)

My default is an 8B or 14B model on an RTX 4090. It’s the cheapest rented card that leaves room for 2,048-token sequences and a reasonable batch, and models in that range are easy to serve afterwards. For picking a base model by the VRAM you’ll serve it on, see which AI models fit your GPU’s VRAM.

Pick a tool: TRL, Unsloth or Axolotl

All three are open source and all do LoRA and QLoRA.

Tool	How you use it	Strength	Watch out for
Hugging Face TRL + PEFT	Python (SFTTrainer)	The reference implementation; DPO, GRPO and more on the same API	Uses more memory than Unsloth on the same run
Unsloth	Python, or the Unsloth Studio web UI	Claims 2x faster and 70% less VRAM; exports straight to GGUF	Studio UI is AGPL-3.0 (core is Apache 2.0)
Axolotl	One YAML file, axolotl train config.yml	Multi-GPU (FSDP, DeepSpeed), many recipes	Needs Python 3.11+ and PyTorch 2.11+

TRL is at version 1.14 and PEFT at 0.21 as of September 2026. Unsloth needs Python 3.11 to 3.13 and an NVIDIA GPU with CUDA capability 7.0 or newer (V100, T4, RTX 20 series and up). Axolotl recommends Python 3.12 and PyTorch 2.12.1.

Use TRL if you want to understand every line, Unsloth if you’re short on VRAM or want GGUF export with one call, and Axolotl if you’ll repeat runs with different settings or move to several GPUs. The script below uses TRL, because it’s the shortest path that shows every moving part.

Prepare the data

TRL’s SFTTrainer reads conversations in the same shape as a chat API request. One JSON object per line in train.jsonl:

{"messages": [{"role": "system", "content": "Extract the invoice fields as JSON."}, {"role": "user", "content": "Invoice 4471 from Norden AB, due 12 March, total 1,250 EUR"}, {"role": "assistant", "content": "{\"invoice_id\": \"4471\", \"supplier\": \"Norden AB\", \"due\": \"2026-03-12\", \"total\": 1250, \"currency\": \"EUR\"}"}]}

Practical rules:

• Quality over count. A few hundred to a few thousand consistent, correct examples beat tens of thousands of noisy ones. Every mistake in the data is behavior you’re paying to teach.
• Match production. Use the system prompt and input format your application will actually send.
• Hold out 5 to 10%. Keep examples the model never trains on, to compare the base model and the tuned one side by side.
• Remove what you don’t need. Names, emails, account numbers and IDs rarely help the model learn a format. Replace them with realistic placeholders before the data leaves your computer.

That last rule is about more than the rented machine. Carlini et al. extracted hundreds of verbatim training sequences from GPT-2, including names, phone numbers and email addresses, some of which appeared in only one training document. A fine-tuned model can repeat what it was trained on to whoever uses it later.

Keep the data private on a rented machine

On a GPU marketplace, someone else owns the computer. Vast.ai puts it plainly: “Clients are isolated in unprivileged Docker containers and only have access to their own data”, and “provider security varies significantly”. That isolation protects you from other renters. It doesn’t protect you from the person with physical access and root on the host.

For private data:

1. Pick a vetted datacenter host. Vast.ai’s Secure Cloud providers are “vetted datacenters with ISO 27001 certification, Tier 3/4 datacenter standards”, and Vast recommends them for sensitive work. RunPod’s Secure Cloud runs in T3/T4 data centers; its Community Cloud connects you to individual providers. The datacenter tiers cost more per hour and are worth it here.
2. Upload only the cleaned dataset, over SSH (rsync -avP or scp). Don’t stage it in a public bucket or a shared link on the way.
3. Keep logging local. In TRL 1.14 report_to defaults to "none", so nothing goes to an experiment tracker unless you turn it on. Don’t call push_to_hub with an adapter trained on private data.
4. Take the results out, then delete the instance. Download the adapter and evaluation outputs, log out of Hugging Face (hf auth logout) if you used a token, and delete the instance and any volume. On Vast.ai, storage is billed and kept until the instance is deleted, not just stopped.

Deleting files inside a container doesn’t guarantee the host’s disk is wiped, so the real protection is steps 1 and 2: choose who holds the hardware, and send them as little as possible. More detail in how to secure a dataset on a public GPU node. If your policy forbids any third-party hardware, the same script runs on your own 24 GB card.

Train: a QLoRA script with TRL

On a rented Linux machine with an RTX 3090 or 4090:

python -m venv venv && source venv/bin/activate
pip install torch trl peft bitsandbytes datasets

Then train.py, following the QLoRA pattern in TRL’s PEFT documentation. Qwen3-8B is Apache 2.0 and not gated, so no Hugging Face token is needed:

import torch
from datasets import load_dataset
from peft import LoraConfig
from transformers import BitsAndBytesConfig
from trl import SFTConfig, SFTTrainer

dataset = load_dataset("json", data_files="train.jsonl", split="train")

bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16,
    bnb_4bit_use_double_quant=True,
)

peft_config = LoraConfig(
    r=16,
    lora_alpha=32,
    lora_dropout=0.05,
    target_modules="all-linear",
    task_type="CAUSAL_LM",
)

args = SFTConfig(
    output_dir="out",
    num_train_epochs=3,
    per_device_train_batch_size=4,
    gradient_accumulation_steps=4,
    learning_rate=2e-4,
    lr_scheduler_type="cosine",
    warmup_steps=20,
    max_length=2048,
    bf16=True,
    logging_steps=10,
    save_strategy="epoch",
    model_init_kwargs={"dtype": torch.bfloat16},
)

trainer = SFTTrainer(
    model="Qwen/Qwen3-8B",
    args=args,
    train_dataset=dataset,
    quantization_config=bnb_config,
    peft_config=peft_config,
)
trainer.train()
trainer.save_model("out/adapter")

The choices that matter:

• learning_rate=2e-4. TRL’s docs recommend about 10 times the normal fine-tuning rate for QLoRA. If the evaluation loss rises while training loss falls, you’re overfitting: use fewer epochs.
• r=16, target_modules="all-linear". Adapters on every linear layer, the setup Unsloth’s benchmarks use. Rank 16 is enough for format and style; raise it for harder tasks.
• max_length=2048. Longer examples are cut. Check your data’s token lengths; a longer limit needs more VRAM.
• Effective batch 16 (4 × 4 accumulation steps). If you run out of memory, lower per_device_train_batch_size and raise accumulation to keep the product.

Before you shut the machine down, run your held-out examples through the base model and the tuned one and compare them. That’s the only test that tells you whether the money did anything.

What it costs

Training time is total tokens ÷ throughput. GigaGPU, a hosting company, published a measured ~3,500 training tokens per second for Llama 3.1 8B with QLoRA on an RTX 4090. Assuming a similar rate for Qwen3-8B:

Small run: 2,000 examples × 600 tokens × 3 epochs = 3.6 million tokens. 3,600,000 ÷ 3,500 = 1,029 s, about 17 minutes.

Step	Time
Set up the environment	10 min
Download Qwen3-8B (16.4 GB of weights) and upload data	10 min
Training	17 min
Compare base and tuned model on held-out data	15 min
Merge, export, download, delete the instance	15 min
Total	67 min (1.12 h)

• Vast.ai RTX 4090 at $0.31/h: 1.12 × $0.31 = $0.35
• RunPod RTX 4090 at $0.74/h (pricing page list price): 1.12 × $0.74 = $0.83

Larger run: 20,000 examples × 1,000 tokens × 2 epochs = 40 million tokens ÷ 3,500 = 11,429 s, about 3.2 hours. With 50 minutes of the same overhead, 4.0 hours: $1.24 on Vast.ai or $2.97 on RunPod.

For a 32B model, 48 GB cards are listed on RunPod at $0.49/h (A40), $0.53/h (RTX A6000) and $1.09/h (L40S) as of September 2026. I don’t have a published throughput for 32B QLoRA on those cards, so run 50 steps, read the step time from the log, and do the same multiplication before you commit to a long run.

Prices are the September 2026 figures from RunPod’s pricing page and getdeploying.com’s tracker for Vast.ai. Secure/datacenter tiers cost more than the cheapest community offers. The broader picture is in GPU rental pricing compared.

Serve the result

You have two options: keep the adapter separate, or merge it into the model.

Keep it separate with vLLM. vLLM loads LoRA adapters next to the base model and exposes each one as a model name on its OpenAI-compatible server:

vllm serve Qwen/Qwen3-8B --enable-lora --lora-modules invoices=./out/adapter

Clients then send "model": "invoices". Several adapters can share one base model on one GPU.

Merge and run it in Ollama. Merge the adapter into full-precision weights, convert to GGUF with llama.cpp, quantize, and import:

import torch
from peft import AutoPeftModelForCausalLM
from transformers import AutoTokenizer

model = AutoPeftModelForCausalLM.from_pretrained("out/adapter", dtype=torch.bfloat16)
model.merge_and_unload().save_pretrained("merged")
AutoTokenizer.from_pretrained("Qwen/Qwen3-8B").save_pretrained("merged")

python llama.cpp/convert_hf_to_gguf.py merged --outfile invoices-bf16.gguf --outtype bf16
./llama.cpp/build/bin/llama-quantize invoices-bf16.gguf invoices-Q4_K_M.gguf Q4_K_M
echo "FROM ./invoices-Q4_K_M.gguf" > Modelfile
ollama create invoices -f Modelfile

Unsloth does the merge and GGUF export in one call (model.save_pretrained_gguf("dir", tokenizer, quantization_method="q4_k_m")). Its docs warn that the most common cause of bad answers after export is the wrong chat template: serve with the template you trained with. The trade-offs between Ollama, vLLM and TGI are in our RTX 4090 inference benchmark.

Where GPUFlow fits

GPUFlow can’t do the training: it rents an OpenAI-compatible API on a provider’s GPU, with no shell, SSH or file access. It also can’t serve your fine-tuned model. Renters can’t upload models; the models on offer are the ones each provider has installed (usually with Ollama), such as qwen2.5:7b or llama3.1:8b.

Where it can help is the step before all this: checking, for a few cents, whether a stock open model with a good prompt already does the job, which is the cheapest outcome in the decision tree. Use test data for that, not the private data this guide is about: prompts and answers pass through the provider’s machine in plaintext while the rental runs. How it works is in the API quickstart, and using the key in apps covers connecting it to existing tools.

Sources

All checked in September 2026.

• Papers: Hu et al., LoRA; Dettmers et al., QLoRA; Ovadia et al., Fine-Tuning or Retrieval?; Carlini et al., Extracting Training Data from Large Language Models
• Hugging Face TRL: SFT Trainer, PEFT integration and QLoRA
• Unsloth: requirements and VRAM table, benchmarks, saving to GGUF, GitHub
• Axolotl on GitHub
• Model: Qwen3-8B model card
• Training throughput: GigaGPU, fine-tuning on the RTX 4090
• Hosts and security: Vast.ai security FAQ, Vast.ai pricing, RunPod Pods overview
• Prices: RunPod pricing, getdeploying.com for Vast.ai and RTX 4090
• Serving: vLLM LoRA adapters, llama.cpp quantize, Ollama import
• GPUFlow: API quickstart