Self-hosted LLM or cloud API

Choosing between a self-hosted LLM and a cloud API is a risk and cost decision, not a technology preference. Cloud APIs give you the strongest models with zero maintenance burden. Self-hosting on infrastructure located in Turkey keeps the data in the country, so the KVKK cross-border transfer regime is never triggered in the first place. In practice one question decides it: can this data leave the country or not? The rest is a table.

Two architectures, two risk profiles

On-premise (self-hosted) LLM means the model weights sit on hardware you control and inference happens inside your own network. Neither the prompt nor the response crosses your perimeter. The hardware may live in your own data centre, a colocation rack, or a dedicated GPU server you rent — in that last case “on-prem” is technically the wrong word, but the question that matters stands unchanged: where is the machine, physically? The legal trigger is not data leaving your network, it is data leaving the country. Self-hosting in a rack abroad gives you no less sovereignty than a cloud API, but it does not get you out of the KVKK transfer regime either.

Cloud API means the prompt travels over HTTPS to the provider’s data centre and is processed there. You don’t hold the model, you hold access to it. The weights are closed, the release calendar is theirs, the price list is theirs.

Both are legitimate. What isn’t legitimate is picking one on ideological grounds and then ignoring the consequences.

Comparison table

Dimension	Self-hosted / on-prem	Cloud API (OpenAI, Anthropic, etc.)
Privacy / data sovereignty	Data never crosses the network boundary. You trust topology, not paperwork. In an audit you can point at a machine	Data goes to the provider. Protection is contractual: OpenAI retains API inputs for up to 30 days by default for abuse monitoring, Anthropic deletes API inputs and outputs within 30 days; zero-data-retention (ZDR) agreements are available for qualifying use cases
Cost model	Fixed cost: GPU capex or monthly server rental, plus power and engineering. Unit cost falls as usage rises — and you pay while it idles	Variable cost, per token. Zero usage, zero invoice. The bill grows linearly with volume
Latency	No round trip to an overseas data centre. But a modest GPU may generate tokens more slowly than a provider’s optimised serving stack	Geographic distance shows up in time-to-first-token; in exchange, throughput is high and consistent
Scaling	Concurrency is capped by hardware. New capacity means new cards and a procurement lead time measured in weeks. Load spikes queue	Elastic. Scales instantly within the provider’s quotas and rate limits
Maintenance burden	Yours: drivers, CUDA, the serving layer (vLLM/TGI/Ollama), model updates, monitoring, redundancy. This has to be part of someone’s actual job	Theirs. Your job is pinning versions and tracking changes to the API contract
Model quality	Open-weight models have closed much of the gap, but frontier models still lead on the hardest reasoning and long-context work. In exchange you’re free to fine-tune on your data and freeze the model	Immediate access to the strongest models. In exchange, the model can be updated underneath you and behaviour can shift quietly
Compliance (KVKK)	If the hardware sits in Turkey, no personal data leaves the country, so Article 9 of Law 6698 is never engaged. Your data controller obligations, of course, remain	This is a cross-border transfer. Article 9, as amended by Law 7499, sets three tiers: adequacy decision → appropriate safeguards → incidental cases. As of publication the Board has issued no adequacy decision, so the practical route is the standard contract — which must be notified to the Authority within five business days of signature

Hardware: realistic expectations

The fastest way to cut through the hype here is arithmetic. The dominant VRAM cost is the weights, and the maths is simple: parameter count × bytes per parameter.

FP16/BF16 → 2 bytes per parameter
8-bit → ~1 byte
4-bit → ~0.5 bytes (quantisation scales and zero-points add a little on top)

Which gives:

Model size	FP16 weights	4-bit weights	What that means in cards
~8B	~16 GB	~4-5 GB	At 4-bit, a single 8-12 GB consumer card is enough
~70B	~140 GB	~35-40 GB	At 4-bit it fits a single 48 GB card (e.g. L40S); FP16 needs 2×80 GB

Weights aren’t the whole story. On top come the KV cache, activations and serving overhead — and the KV cache grows with concurrent requests and context length. So “the model fits on the card” and “twenty people can work on long documents at once” are very different claims. Plan capacity against concurrency × context length, not parameter count.

Let’s be honest about the trade: quantisation isn’t free. Four-bit shrinks the weights but can cost measurable output quality, and how much depends on the model and the task. You measure that with your own evaluation set, not with a guess.

When on-prem wins

When the data category simply cannot leave the country: health records, legal files, HR dossiers, defence work, unreleased financials
When you don’t want to carry — or can’t carry — the standard contract and notification burden for cross-border transfer
When volume is high, steady and predictable; fixed-cost economics start working for you
When you need to freeze behaviour, so tomorrow’s model release doesn’t change today’s output
When the workload is narrow and repetitive: classification, extraction, summarisation. A small open model, fine-tuned, is more than enough here

When a cloud API wins

When you’re still validating the idea. Buying GPUs before the idea lands is the most expensive way to learn
When the work needs frontier reasoning, multi-step agentic behaviour or very long context
When volume is low or spiky
When there’s no in-house engineering capacity to operate an LLM service — admitting this plainly is a virtue, not a weakness
When the data isn’t personal data at all: public content, product catalogues, anonymised text

Hybrid: the honest answer for most organisations

The question is usually not “which one” but “which data goes where”. The pattern that works: sensitive workloads hit an open-weight model inside the perimeter, while non-sensitive or reasoning-heavy workloads route to a cloud API. The routing decision lives in the application layer, driven by a rule that reads your data classification. The RAG systems we build split especially cleanly this way: the document index and embedding generation stay in-house, and you only reach outside when you must — and then with masked inputs.

Three questions to decide

Can this data leave the country? If no, the discussion is over — on-prem, on hardware in Turkey.
Is your monthly token volume predictable? If no, start with an API, measure, then decide.
Who keeps this running six months from now? If you can’t name a person, don’t go on-prem.

How we help

Data sovereignty isn’t a marketing line for us, it’s a service we deliver: running open-source or closed models on the client’s own servers, and designing hybrid setups with cloud APIs where that’s the right call. We look at your data classification and your real volume first, and decide the hardware after — because doing it the other way round gets expensive. If you’d like to talk: contact us or [email protected].