What a local model can actually do: 75 business tasks, measured

A clinic or a workshop wants an assistant for the same ten questions in WhatsApp and Telegram: price of a cleaning, is there a slot tomorrow. The client file should not leave the room. The honest question is not “is local AI the future.” It is: will an office machine do the job, and will it invent a discount that does not exist?

The way to answer that is a measurement, not a demo.

What was run

An ordinary strong workstation: NVIDIA RTX 3090 (24 GB), 64 GB of RAM. Two open-weight models. Seventy-five office tasks, five families, fifteen items each:

1. Turn an inquiry into fields. A messy booking message into name, phone, service, preferred window.
2. Answer from a document. A price list or a policy, a question whose answer is in the text.
3. Refuse when the answer is not there. Price of a service the list does not carry, a street address the file never states.
4. Classify the turn. Booking, complaint, price question, general, spam.
5. Choose a tool. Book, cancel, quote, check openings.

A person did not grade “by eye.” The model did not grade itself. A script compared each answer to a frozen gold set. Thresholds were locked before the first run.

Public benches for these models are English leaderboards. An operator needs the same discipline on the tasks the desk actually runs.

What held

The primary subject was a compressed Qwen (17 GB on disk). After the instruction was set correctly:

Median latency on short tasks: 2.6 to 5.5 seconds. No dropped connections.

Fifteen items per family is not a guarantee for every future sentence. It is enough to show that, on these jobs, the program is usable.

Will it lie to a customer

The business fear is invented prices and invented services. Fifteen questions whose answers were absent from the file: a treatment not on the list, headcount, whether instalments exist.

Result: 100% correct refusals. Zero invented figures. The assistant said, in substance, that the document names a city and does not contain a street or a route.

That is not the whole story.

The same model, a slot-check task, no “today” in the prompt: “Is anything free tomorrow?” It checked a calendar date it had made up. Nobody had passed the current day. Instead of saying it did not know, it supplied a plausible one.

One line in the instruction

The difference between a clean refusal and a invented date was not the weights. It was the brief.

The first family said: if it is not there, say so. The slot family did not. “I do not know” is not a native virtue. It is a rule you write.

The system prompt was changed: pass today’s date, and add “if the data is missing, leave the field empty, do not invent.”

Result: a clean set. It stopped inventing the date and started resolving “Friday” and “3 March” against the calendar it had been given. No new GPU. No fine-tune. One line.

A high score on a model card and a working assistant are different objects. The setting sits between them.

How much text it can hold

Dialogue memory is physical. The 17 GB file plus the context window compete for 24 GB. At moderate documents the card is already past eighty percent; a long thread left 628 MB free.

Latency against document size:

Who is waiting matters.

A live chat has a ten-second ceiling. Nobody sits for two minutes. On this machine that is 15–20 pages: a price list, a policy, a staff brief — with margin.

A batch job — overnight inbox, a dump, a report — can take two minutes. The same 120 pages are fine.

One bench, two conclusions. The easy mistake is to apply the chat budget to every job and declare the box too weak.

Size the window to the documents you have. A giant reservation “just in case” makes a short question twice as slow. The pocket of spare context has a cost.

A memory-compression flag, on this pair of models, did not cost quality, and it freed almost two gigabytes on a long thread.

Two models, two manners

The second file was lighter (about 14.8 GB). On extraction, price-list answers, and refusals, both cleared the bar.

They differed when data was missing. On “Is tomorrow free?” the lighter model stopped and asked which date “tomorrow” is. Qwen, in the same seat, sometimes invented a date and sometimes asked — even with settings frozen. Boundary behaviour drifts inside one configuration.

Asking is the correct default. A booking on a random day is more expensive than one extra turn. Manner is a product choice. The rule “if data is missing, ask” is not left to the model.

Thinking depth, left on maximum, can look wasteful on a single item (hundreds of tokens of work for “where are you?”). Across all 75 tasks, turning it down saved about eight percent — and shifted spend the wrong way on the simple items. Quality did not drop; on two families it improved. Defaults are not your settings. Measure them.

Speculative decoding

A small draft model proposes; the main model checks. Quality did not move by a tenth on either file. The lighter model finished the 75-task set in 160 seconds instead of 379 (2.4×). Qwen: 271 down to 170 (1.6×). The tax is VRAM for the drafter (about 2.5 GB and 1.3 GB). Near-free speed if the card has the room.

Wall-clock, the two stacks then sit together (170 s vs 160 s). Qwen still answers more tightly. If one configuration has to be chosen for desk work, that is the one this bench would keep.

Wording is not bit-stable. “Cancel at least four hours ahead” and “no later than four hours before the appointment” are both right. Quality can be trusted. Verbatim identity cannot.

What it means on the floor

A local assistant is a tool: classify, answer from a policy, check a grid — on a machine in the room, with client data that does not leave.

A downloaded file is not a colleague. Without the rule, it will fill gaps. Context has to be sized. The privacy invoice — own box, rental GPU, or API — is a separate measurement.