Local models vs API: compare workload, operating cost and data residency

Compare a local model, a rented GPU and a hosted API against the same workload before deciding which costs less. First exclude options that cannot meet the data-handling requirement. Then compare accepted outputs, latency, utilisation and operating effort rather than an isolated token rate or hardware invoice.

This article does not publish tariffs, hardware prices or payback estimates. Those inputs change and require dated quotes for the chosen configuration. The worksheet below keeps the comparison reproducible without pretending one deployment path is always cheapest.

Define the workload before the machine

A model loading successfully is only one acceptance condition. Test the actual prompt length, retrieval context, output length and concurrent requests. Record the model version, quantisation, runtime and hardware or service configuration. Memory needs depend on those choices; 24 GB is not a universal minimum for local inference.

Separate an interactive assistant from a batch job. The assistant needs an agreed response-time target under expected load. Batch work may tolerate a later result, but still needs an output-quality check and a recovery plan. Do not compare their throughput as if the service promise were identical.

Use a representative, approved test set. Record rejected answers and retries as well as accepted ones. A fast output that fails the business check is not an equivalent unit of work.

Compare three operating models

Scroll horizontally to compare all columns on a narrow screen.

For accelerator support, check the current runtime and model documentation for the specific device. Do not assume either that all inference requires CUDA or that equal advertised memory guarantees equal performance across devices.

For used hardware, request evidence of operation under a representative sustained workload, memory errors, cooling condition and the supplier’s support terms. One short stress test is not a guarantee of remaining service life. Use the manufacturer’s operating limits for that device rather than a generic temperature threshold.

Keep the comparison on one basis

Comparison date / observation window:
Workload version / expected volume / concurrency:
Quality acceptance rule / reviewer:
Response-time requirement:
Data-residency and retention constraints:

For each option:
Model / runtime / hardware or API version:
Accepted outputs / attempted outputs / retries:
Measured latency / utilisation / test conditions:
Quoted service or acquisition assumptions / date:
Energy / storage / transfer / maintenance assumptions:
Operator effort / recovery responsibility:
Unmeasured inputs / reason:

Compare total operating assumptions over the same period and divide by accepted work only when the quality rule is comparable. Keep one-time setup and recurring effort distinguishable. If a cost input is unknown, leave it unknown rather than filling it with a market average.

A payback estimate needs a stated comparison period, useful-life assumption and workload. Test how it changes when demand, idle hours or maintenance effort changes. A result for continuous operation does not establish the result for occasional office use. This is a proposed decision method, not a newly measured studio benchmark.

Local execution is only one privacy boundary

Running model inference locally can keep that inference request on the chosen machine. It does not establish that the whole assistant is local. Embeddings, document parsing, logs, telemetry, backups, support tools and external integrations can still send data elsewhere.

Map each transfer and its purpose. Review access, retention and service terms with the responsible owner. A rented GPU is remote infrastructure even when the runtime is self-managed. A hosted API’s suitability depends on the applicable agreement and controls, not only its billing unit.

If documents cannot leave an approved environment, test the entire path against that requirement before comparing cost. If external processing is permitted, evaluate it alongside the operational burden of self-hosting. Ownership alone does not guarantee privacy, and an API alone does not establish an unacceptable risk.

Make the decision from evidence

Keep options that meet the acceptance, data and recovery requirements. Compare their measured workload and dated operating assumptions, then name the remaining uncertainty. A smaller deployment may be enough; a demanding workload may need a different model or more capacity. Neither conclusion follows from the label local or API.

Start with a written brief: the data constraint, representative tasks, live versus batch timing and who will operate the system. The comparison should end with a testable deployment choice and named recovery owner.