Runs every day on GitHub Actions. Samples labelled messages (synthetic, not customer data), runs the agent (claims guardrail, then an LLM classifier with a local fallback, then a confidence router) and scores it. Last run: 2026-11-11 · mode: llm · model: gemini-3.1-flash-lite
| Date | Mode | Acc. familiar | Acc. new wording | Auto acc. (new) | Auto rate | Claims to human |
|---|---|---|---|---|---|---|
| 2026-11-11 | llm | 96.0% | 80.0% | 92.3% | 56.0% | 100.0% |
| 2026-11-10 | llm | 92.0% | 72.0% | 83.3% | 44.0% | 100.0% |
| 2026-11-09 | llm | 96.0% | 92.0% | 92.9% | 48.0% | 100.0% |
| 2026-11-08 | llm | 88.0% | 76.0% | 90.0% | 68.0% | 100.0% |
| 2026-11-07 | llm | 92.0% | 80.0% | 88.9% | 60.0% | 100.0% |
| 2026-11-06 | llm | 96.0% | 72.0% | 69.2% | 60.0% | 100.0% |
| 2026-11-06 | fallback-only | 97.5% | 20.0% | 0.0% | 52.5% | 100.0% |
| Message | True | Predicted | Conf. | Source |
|---|---|---|---|---|
| when is my next payment due also our driver rear-ended someone | billing | claim_report | 1.0 | guardrail |
| thanks need the acord 25 form also one of our trucks was in an accident | certificate_request | claim_report | 1.0 | guardrail |
| got a late payment notice also need the acord 25 form | billing | certificate_request | 0.9 | llm |
| thanks is there a fee for paying in installments also we bought another van | billing | policy_change | 0.9 | llm |
| following up - client won't pay until they get our insurance certificate also closing the business, please cancel coverage | certificate_request | cancel | 0.95 | llm |
| following up - please terminate effective end of month also employee got hurt on the job today | cancel | claim_report | 1.0 | guardrail |
Source: github.com/f20220970-debug/llm-triage-agent