AI search and chatbot acceptance testing
A written protocol for 200 resident queries, multilingual scoring, prompt-injection testing and a reviewable release decision.
Newton & Carter. Published November 7, 2026; updated November 9, 2026. Version 1.1.
Approve the question and source set
Client domain and language reviewers approve a frozen corpus, 200 expected questions or refusals, and the scoring rubric. The proposed worksheet allocates 50 cases each to English, Spanish, Vietnamese and Tagalog.
Separate accuracy from safety
The proposed accuracy gate is at least 190 correct of 200. Report each language separately and agree its minimum before running. A separate prompt-injection, jailbreak and tool-boundary track requires zero critical safety failures.
Preserve the run and human decision
Version the system, sources, prompts, model, settings and permissions. Retain every result and failure, use independent domain/language review, rerun after changes and record client acceptance. The public worksheet is blank and has not been run.
Version 1.1 · Published November 9, 2026 · Not run
Agree the test before measuring the answer.
This protocol provides a blank 200-case benchmark and a separate safety test plan. Client-approved sources, language review and acceptance sign-off are required before a result can be accepted.
Proposed acceptance gates — targets, not measured results
- Accuracy
- At least 190 of 200 correct (95%) on the frozen set.
Current result: Not run
- Language review
- Report each language separately and meet its floor, agreed before execution.
Floors: Not agreed
- Safety
- At least 40 separate tests and zero critical safety failures. Excluded from accuracy scoring.
Current result: Not run
These numerical gates are proposed for this protocol. NIST and OWASP do not prescribe them; passing a benchmark does not guarantee production safety.
| Language | Cases | Status |
|---|---|---|
| English (EN) | 50 | Not run |
| Spanish (ES) | 50 | Not run |
| Vietnamese (VI) | 50 | Not run |
| Tagalog (TL) | 50 | Not run |
Freeze, run, review and retain the evidence.
- Approve the cases. Client domain and language reviewers approve the corpus, query mix, expected answers or refusals, and supporting sources. Only a prelabelled expected refusal can count correct; false refusals are incorrect.
- Version the run. Freeze the application, corpus, prompts, model, settings, permissions, language pipeline, case set and scoring rubric. Record expected source, actual answer, correctness and critical errors for every case.
- Review independently. Two qualified domain/language reviewers grade anonymized outputs blind to the system variant. An independent adjudicator resolves disagreement and preserves both original grades. Report overall and per-language results, false refusals and critical errors.
- Test safety separately. Cover direct and indirect injection, jailbreaks, synthetic-canary exfiltration and tool boundaries in supported languages. Use authorized isolated environments and instrumented tool substitutes. Any critical failure blocks release, regardless of accuracy.
- Retest after changes. Rerun the full frozen accuracy and safety sets after source, prompt, model, setting, permission or application changes. Run a separate unseen holdout with pre-agreed size, language coverage and floors. Retain failures and every repeat; obtain client sign-off.
What the worksheet and run record contain
The CSV has exactly 200 blank accuracy rows. Only case ID, language and not-run
status are filled. Record the query, expected disposition, expected answer/source and source version, actual answer/sources, correctness, critical error, two reviewers, adjudication and run ID.
The JSON adds run/version fields and 40 blank safety starters: eight in each of the five safety families. Safety and holdout results never enter the 200-case score. Missing outputs, timeouts and unresolved grades cannot be dropped or counted correct. Incomplete runs remain incomplete; language floors cannot be overridden by an aggregate pass.
Use a pre-agreed repeat policy for nondeterministic output. Keep original failures and all runs; do not choose the best repeat or tune to an exposed holdout and call it unseen. Canary non-disclosure only evaluates the tested marker and path.
Download the method and blank records.
- Download the one-page method (PDF)
- Download the complete method (Markdown)
- Download the blank 200-case worksheet (CSV)
- Download the method and blank accuracy/safety records (JSON)
The CivicAnswer evaluation workbench is a 50-case deterministic demonstration. Read its published scope and limits. Its results apply to that versioned example dataset. This protocol requires a new client-approved dataset and signed run record.
Primary references: NIST AI RMF — Measure; OWASP Prompt Injection; OWASP prevention and testing guidance. These inform documentation, independent review and boundary testing; they do not certify a system.
References for the work
Use these published resources when defining the requirements for your project.
- NIST AI Risk Management Framework Playbook ↗
Review the Measure function when defining evaluation evidence, methods and limitations.
Run the agreed protocol with your sources.
Bring the source corpus, supported languages, intended tasks and release criteria. The engagement defines implementation, evaluation ownership, retained evidence and the client acceptance decision.
Download the one-page acceptance method ↗
Discuss the evaluation scope ↗