AI Agents: Acceptance criteria

Detailed practical guide

Prepare your workspace and inputs

Create a local text file called acceptance_lab.py. The fictional incident begins at 09:00 and recovers at 09:20 UTC. The incident owner is unknown. Save the script in step 2 and run python3 acceptance_lab.py, or py -3 acceptance_lab.py on Windows.

Platform context: Python 3.10+; standard library only; local script. No model or external service is required.

Scenario to reproduce or investigate

An agent returns a convincing incident summary but omits an event, fabricates an owner or fails to cite evidence. A text response by itself must not complete the task.

1. Write a machine-checkable output contract

   Represent the expected artifact as a dictionary with timeline, owner and unresolved fields. Require the two event IDs E1 and E2 exactly once and require each event time to match the supplied source. Owner must remain null because neither source names an owner. Record these requirements before accepting any output.

   Command, configuration or input

   Expected source events: E1=09:00Z (outage), E2=09:20Z (recovered)
   Expected owner: null
   Required unresolved item: cause

   Expected result · ผลที่ควรเห็น

   The requirements can be checked without asking the same model whether its own answer is correct.

   If it fails · เมื่อไม่ผ่าน

   If a criterion contains only terms such as useful or high quality, replace it with the required field, source relationship or allowed value.

2. Implement the validator and exercise three responses

   Save this entire script. It accepts the complete source-grounded response and rejects both an omitted recovery event and an invented owner. The application owns the source map; do not let a model replace the expected facts.

   Save and run this example

   from copy import deepcopy
   
   SOURCE = {"E1": "09:00Z", "E2": "09:20Z"}
   
   def validate(result):
       errors = []
       if not isinstance(result, dict):
           return ["result must be an object"]
       rows = result.get("timeline")
       if not isinstance(rows, list):
           return ["timeline must be a list"]
       if any(not isinstance(row, dict) for row in rows):
           return ["timeline entries must be objects"]
       ids = [row.get("source_id") for row in rows]
       if any(not isinstance(value, str) for value in ids):
           return ["source IDs must be strings"]
       if len(ids) != len(SOURCE) or set(ids) != set(SOURCE):
           errors.append("missing, duplicate or unknown event")
       for row in rows:
           if row.get("time") != SOURCE.get(row.get("source_id")):
               errors.append("event time does not match source")
       if result.get("owner", "missing") is not None:
           errors.append("unsupported owner")
       if not isinstance(result.get("unresolved"), list) or "cause" not in result["unresolved"]:
           errors.append("unknown cause must remain unresolved")
       return errors
   
   good = {"timeline": [{"source_id": k, "time": v} for k, v in SOURCE.items()],
           "owner": None, "unresolved": ["cause"]}
   missing = deepcopy(good)
   missing["timeline"].pop()
   invented = dict(good, owner="Pat")
   assert validate(good) == []
   assert validate(missing)
   assert validate(invented)
   assert validate({"timeline": "not a list"})
   print("PASS: complete accepted; omitted, invented and malformed results rejected")
   

   Expected result · ผลที่ควรเห็น

   The script prints PASS: complete accepted; omitted, invented and malformed results rejected and exits with status 0.

   If it fails · เมื่อไม่ผ่าน

   An AssertionError identifies a gap in validation. Preserve the failing response and fix the corresponding contract check before integrating a model.

3. Put the check at the completion boundary

   In the application, parse the candidate output, enforce a request-size limit, run validate and write the final artifact only when the returned error list is empty. Return validation errors to the repair step with a bounded retry count. Keep the original candidate and source revision for audit. Add a test in which the model says complete while the dictionary is still missing E2.

   Command, configuration or input

   errors = validate(candidate)
   if errors:
       status = "needs_repair"
   else:
       status = "ready_for_review"

   Expected result · ผลที่ควรเห็น

   Completion depends on source checks, not a model-selected done flag; an unresolved cause can coexist with an accurate timeline.

   If it fails · เมื่อไม่ผ่าน

   If the artifact is written before validation, move the write behind the success branch. If repeated repairs fail, return an explicit incomplete status rather than looping indefinitely.

Recovery and cleanup

This lab changes no external system. Delete the local script when finished, or retain it as a regression test. In an application, restore the previous validator behind the deployment rollback if new checks reject previously supported input shapes.

Apply to actual work and known limits

Replace SOURCE with independently supplied incident evidence and extend the contract to your actual artifact schema. This validator does not establish that free-text causal claims are true; review or constrain those claims separately.

The included self-contained Python fixture was executed successfully in this Linux environment. External products and production infrastructure were not exercised.

PASS: complete accepted; omitted, invented and malformed results rejected