LiveOps Agent Eval

Simulated casual puzzle game (60 levels, 27,000 players, 75 days). Each policy faces the same players and the same random draws. Not real game data. Source: github.com/f20220970-debug/chaos-liveops-agent

Policies compared (mean of 3 seeds)

Policy	D7 retention	vs no agent (pts)	D14 retention	Revenue / install	vs no agent	Changes shipped	Monetisation levels eased
No agent (designer's curve)	51.6%	+0.0	25.4%	$0.495	+0.00%	0.0	0.0
Naive agent (no tests, no guardrails)	56.5%	+4.9	30.3%	$0.493	-0.27%	17.3	1.0
Guarded LiveOps agent	52.7%	+1.1	26.7%	$0.494	-0.05%	2.3	0.0

Daily runs (one new simulated world per day)

Date	Seed	Guarded: D7 change (pts)	Guarded: revenue change	Naive: revenue change
2026-11-11	9931	+2.7	-0.27%	+0.40%
2026-11-10	9930	+1.9	+0.50%	-0.22%
2026-11-09	9929	+1.3	+0.45%	+1.22%
2026-11-08	9928	+2.3	+0.49%	+0.42%
2026-11-07	9927	+1.8	-0.25%	-0.64%

Guarded agent decision log (seed 0)

Day	Level	Action	Reason	Pass rate control	Pass rate treatment	Revenue/day change
7	9	start_test	churn 2.2x neighbours; testing pass rate x1.3 on 50% of players
14	9	ship	more players pass the level; retention and revenue inside guardrails	0.8205	0.8837	0.0047
22	26	start_test	churn 1.8x neighbours; testing pass rate x1.3 on 50% of players
24	17	start_test	churn 1.8x neighbours; testing pass rate x1.3 on 50% of players
29	34	start_test	churn 2.0x neighbours; testing pass rate x1.3 on 50% of players
32	26	ship	more players pass the level; retention and revenue inside guardrails	0.8669	0.9232	-0.0575
32	17	rollback	players pass more, but a guardrail was breached (revenue or retention)	0.875	0.9212	-0.1135
37	43	start_test	churn 1.9x neighbours; testing pass rate x1.3 on 50% of players
39	17	start_test	churn 1.8x neighbours; testing pass rate x1.3 on 50% of players
40	34	ship	more players pass the level; retention and revenue inside guardrails	0.8101	0.88	-0.0469
47	17	ship	more players pass the level; retention and revenue inside guardrails	0.8799	0.9523	0.1365
52	43	rollback	no significant gain	0.8881	0.9226	-0.1507
53	9	start_test	churn 1.9x neighbours; testing pass rate x1.3 on 50% of players
56	17	start_test	churn 2.1x neighbours; testing pass rate x1.3 on 50% of players
57	51	start_test	churn 2.0x neighbours; testing pass rate x1.3 on 50% of players
68	9	rollback	inconclusive at time limit	0.0	0.0	0.0
71	17	rollback	inconclusive at time limit	0.0	0.0	0.0
72	51	rollback	inconclusive at time limit	0.783	0.8744	0.097

LiveOps agent: evaluation and decision log

Policies compared (mean of 3 seeds)

Daily runs (one new simulated world per day)

Guarded agent decision log (seed 0)