Agent Reliability LabDiscuss your needs
Automation

Model Run: Service correspondence to reviewable work proposals

Synthetic model example of extracting work proposals and local calendar drafts from service dispatch threads, showing measured omissions, fallback artifacts, and required operator review.

Convert multi-message service and dispatch correspondence into structured work proposals, a review report, and local calendar drafts—without sending customer replies or booking live slots automatically.

Note: This page presents a single synthetic model example from our Agent Reliability Lab (measured November 16, 2026 on DeepSeek V4.1 Flash 1017 against reference expectations fixed before the run). It is not a client case study or an overall accuracy score.

Who it is for

Field-service and appointment-based operations managing scheduling, dispatch, and job updates across email or message threads that need structured proposals for dispatcher review before any booking or reply happens.

Input → Output

  • Input: 16 correspondence threads (33 messages total) paired with business rules, a service catalog, and a schedule.
  • Output:
    • 13 raw work proposals
    • Final report (containing 17 order objects—see errors below)
    • 5 local calendar event drafts (zero real bookings, zero outbound messages sent, and zero live calendar imports)

Model Run: Results and Measured Errors

In this November 16 synthetic English evaluation run (DeepSeek V4.1 Flash 1017, reference expectations locked prior to execution):

What matched

  • Raw exact threads: 14/16
  • Final exact threads: 11/16
  • Thread-level routing decisions: 16/16
  • Quoted evidence check (120/120): Every emitted quote literally matched text in the source messages. (Note: this verifies quote accuracy, not whether the reasoning or justification is complete.)

Where the run fell short

  • Missed warning flags (Flags: TP 11 / FP 0 / FN 2): Two required flags were missed—one for low confidence and one for an overdue promise.
  • Extra empty objects in the final report: The final report contains 17 order objects (13 actual proposals plus 4 extra empty fallback objects). These are not 17 valid orders.
  • Unsupported address line: An address line not supported by the raw correspondence was added after the raw extraction stage.
  • Raw boolean discrepancies: Two raw boolean fields disagreed with the fixed reference expectations.

What the operator checks

Because calendar outputs are strictly local drafts and the model can add unsupported fields or drop warning flags, an operator must:

  1. Verify every address line directly against the original thread messages.
  2. Check that all warning flags are present—especially low-confidence extractions and overdue customer commitments.
  3. Remove any empty fallback objects from the final report before downstream handling.
  4. Confirm assigned resource and time slot before creating any real booking, sending a customer reply, or importing to a live calendar.

Limitations

  • No live actions executed: Automatic calendar import, live dispatch booking, and outbound messaging are not enabled or approved. The 5 calendar events produced in this run are local drafts only.
  • Synthetic sample only: Results come from one synthetic English dataset (16 threads, 33 messages). They do not establish an overall accuracy percentage, production reliability, or proven improvement over earlier runs.
  • Untested edge cases: This evaluation scope did not test overnight or ambiguous DST appointments or conflicting duplicate message IDs.

Next step

Want to define how your service threads, rules, and review gates should be structured before any automation touches your schedule?

Start with a written brief

Agent Reliability Lab
AI consultant

AI replies may be wrong. We store conversations in a restricted server log for 30 days to evaluate reply quality and review enquiries. Messages and chat history are sent to an external AI service. Do not share passwords or confidential information. Privacy · Contact the studio directly