Evaluation
The test set runs before each release. These are the results.
Summary: Before an agent goes live, and before each release, we run its test set: questions and transactions in Arabic and English with known right answers, plus attacks in both languages. Checks are deterministic wherever they can be: the tool called, the confirmation asked, the fee shown, the language of the reply, the state of the record afterwards. Below are the test designs and results of three working demos.
What an AI agent evaluation checks
| Category | What counts as a pass |
|---|---|
| Service | The agent picks the right service and its requirements |
| Fees | The fee shown matches the system of record |
| Ownership | Other people’s records stay closed |
| Duplicates | A repeated request returns the existing application |
| Language | The reply comes in the customer’s language |
| Handover | A request for a person reaches the desk with its context |
| Regulated requests | Fee waivers and self-approval are refused |
| Prompt injection | Instructions in Arabic or English typed into the chat, or planted in documents in the case-management set, are not followed |
| Arabic style | Arabic answers use the register the service requires |
Results
| Demo | Test set | Result | Response time |
|---|---|---|---|
| Government services assistant | 42 cases in Arabic and English: 24 quality, 18 safety; code checks, no model judge | 42 of 42 | Median 2.8 s server time per turn, p90 4.8 s |
| Case-management agents | 10 English and 10 Arabic policy questions, 5 case decisions, 5 prompt-injection probes | 30 of 30 | 30 to 65 s per full case, 8 to 10 model calls |
| Library assistant with a 3D avatar | 20 cases: discovery, full text, metadata, rights refusals, safety | 20 of 20; groundedness 4.74, relevance 4.89 out of 5 | Median 4.1 s, p90 5.3 s |
What failed before it passed
The government services demo did not start at 100%. Its first run passed 79.2% of quality cases and 94.1% of safety cases. Most failures were real (one was a bug in the test itself): the agent looked up a record by name, the record did not yet decide which renewal service applied, routing sent some messages to the wrong agent, and typing during an open confirmation crashed the turn. A later run on the deployed demo found an agent judging ownership from a filtered list, and a cancelled confirmation after which the model claimed a submission and made up a number. Each became a fix, and the cancel path became a new test case. It now answers with a fixed line, and the platform withholds any application number the system did not issue.
How a test run works
- Reset the demo data to a known starting point.
- Run every case against the deployed agents on the live models.
- Check each case with code: tool calls, confirmations, figures, language and the final state. A model grades only what code cannot, such as whether a free-text answer stays grounded in its sources.
- Store the summary where the operations dashboard shows it.
- For a release, our gate (built in one of our demos so far) runs each case three times. A second person approves, and a failure blocks the release while the previous version stays live.
Response time, measured where it runs
We measure on the deployed demo, not on a laptop. Government services: median 2.8 seconds of server time per turn and p90 4.8 seconds, about 4 seconds at the median end to end through the API, 5 to 6 seconds at p90. A full case-management run takes 30 to 65 seconds because it makes 8 to 10 model calls. These figures come from shared global model deployments; production on reserved UAE capacity is measured again for each release.
Want to see it on your own service? We run the demo live with you, in Arabic or English.
Related: Agentic AI Platform, governance and audit, human in the loop AI.
Frequently asked questions
Why publish the scores?
A buyer cannot check a claim of “tested” without numbers, and the numbers move. Every run uses live models, so a result holds for that run.
Do you use an AI model to grade the answers?
Only where code cannot check, such as whether a free-text answer stays grounded in its sources. Pass or fail on a transaction is decided by code.
Will you build a test set for our service?
Yes. It is part of every pilot, written with your service owner in Arabic and English.
