What a local model can actually do: 75 business tasks, measured

A clinic or a workshop wants an assistant for the same ten questions in WhatsApp and Telegram: price of a cleaning, is there a slot tomorrow. The client file should not leave the room. The honest question is not “is local AI the future.” It is: will an office machine do the job, and will it invent a discount that does not exist?

The way to answer that is a measurement, not a demo.

What was run

The figures below are retained from the studio’s report dated 23 August 2026. The original run logs, exact model builds and grader were not revalidated in this editorial pass. The figures describe that report, not independently confirmed current performance.

An ordinary strong workstation: NVIDIA RTX 3090 (24 GB), 64 GB of RAM. Two open-weight models. Seventy-five office tasks, five families, fifteen items each:

1. Turn an inquiry into fields. A messy booking message into name, phone, service, preferred window.
2. Answer from a document. A price list or a policy, a question whose answer is in the text.
3. Refuse when the answer is not there. Price of a service the list does not carry, a street address the file never states.
4. Classify the turn. Booking, complaint, price question, general, spam.
5. Choose a tool. Book, cancel, quote, check openings.

A person did not grade “by eye.” The model did not grade itself. A script compared each answer to a frozen gold set. Thresholds were locked before the first run.

An operator needs a frozen evaluation set for the language and tasks the desk actually uses; a public leaderboard does not replace it.

What held

The reported primary subject was a compressed Qwen file. This public summary does not identify exact model builds or the complete runtime configuration, and these runs were not reproduced for this editorial review. Treat the figures as a bounded historical report, not a reproducible benchmark specification. After the instruction change, the reported results were:

Median latency on short tasks: 2.6 to 5.5 seconds. No dropped connections.

Fifteen items per family describe a small observed fixture set. They do not establish reliability for new questions, concurrent users or a different model/runtime build.

Will it lie to a customer

The business fear is invented prices and invented services. Fifteen questions whose answers were absent from the file: a treatment not on the list, headcount, whether instalments exist.

Result: 100% correct refusals. Zero invented figures. The assistant said, in substance, that the document names a city and does not contain a street or a route.

That is not the whole story.

The same model, a slot-check task, no “today” in the prompt: “Is anything free tomorrow?” It checked a calendar date it had made up. Nobody had passed the current day. Instead of saying it did not know, it supplied a plausible one.

One line in the instruction

The difference between a clean refusal and a invented date was not the weights. It was the brief.

The first family said: if it is not there, say so. The slot family did not. “I do not know” is not a native virtue. It is a rule you write.

The system prompt was changed: pass today’s date, and add “if the data is missing, leave the field empty, do not invent.”

Result: a clean set. It stopped inventing the date and started resolving “Friday” and “3 March” against the calendar it had been given. No new GPU. No fine-tune. One line.

A high score on a model card and a working assistant are different objects. The setting sits between them.

How much text it can hold

Dialogue memory is physical. The 17 GB file plus the context window compete for 24 GB. At moderate documents the card is already past eighty percent; a long thread left 628 MB free.

Latency against document size:

Who is waiting matters.

A live chat needs an agreed response-time budget for its audience and task. Ten seconds can be a proposed test budget, not a universal ceiling. Retest document length and concurrency on the chosen setup.

A batch job may tolerate a longer wait. Set its completion deadline and validate document coverage separately; the reported latency alone does not establish acceptable answers.

One bench, two conclusions. The easy mistake is to apply the chat budget to every job and declare the box too weak.

Measure context allocation against the real document and conversation workload. A larger reservation can increase resource use; the effect depends on the runtime and configuration.

The historical report describes a memory-saving configuration on the tested pair. Retest its effect on quality, latency and memory for the selected model rather than treating the reported result as a deployment guarantee.

Two models, two manners

The second file was lighter (about 14.8 GB). On extraction, price-list answers, and refusals, both cleared the bar.

They differed when data was missing. On “Is tomorrow free?” the lighter model stopped and asked which date “tomorrow” is. Qwen, in the same seat, sometimes invented a date and sometimes asked — even with settings frozen. Boundary behaviour drifts inside one configuration.

Asking is the correct default. A booking on a random day is more expensive than one extra turn. Manner is a product choice. The rule “if data is missing, ask” is not left to the model.

Thinking depth, left on maximum, can look wasteful on a single item (hundreds of tokens of work for “where are you?”). Across all 75 tasks, turning it down saved about eight percent — and shifted spend the wrong way on the simple items. Quality did not drop; on two families it improved. Defaults are not your settings. Measure them.

Speculative decoding

A small draft model proposes; the main model checks. Quality did not move by a tenth on either file. The lighter model finished the 75-task set in 160 seconds instead of 379 (2.4×). Qwen: 271 down to 170 (1.6×). The tax is VRAM for the drafter (about 2.5 GB and 1.3 GB). The reported speedup also required memory for the draft model; measure the trade-off on the selected setup.

Wall-clock, the two stacks then sit together (170 s vs 160 s). Qwen still answers more tightly. This historical comparison is insufficient to choose a current deployment without the model identifiers, runtime settings and a representative retest.

Wording is not bit-stable. “Cancel at least four hours ahead” and “no later than four hours before the appointment” are both right. These observed wording differences were acceptable for the fixture; they do not establish general answer quality or exact reproducibility.

What it means on the floor

A local assistant is a tool: classify, answer from a policy, check a grid — on a machine in the room, with a data-flow review covering inference, retrieval, logs and integrations.

A downloaded file is not a colleague. Without the rule, it will fill gaps. Context has to be sized. The privacy invoice — own box, rental GPU, or API — is a separate measurement.