I Built a Game. Well… The Agents Built It With Me
When agents generate the parts, who owns the whole?
A conference session about multi-agent generation, integration labor, and the missing architectural responsibility for keeping generated work coherent.
I built a game using autonomous agents — code, visuals, music, mechanics, iterations. But the real lesson was not that agents can generate parts. The real lesson was that someone still has to own whether those parts add up to one coherent experience.
From generated parts to coherent experiences
I have been exploring a simple question:
What happens when autonomous agents are not just asked to generate isolated outputs, but to help create a complete interactive experience?
- Not just a piece of code.
- Not just an image.
- Not just a soundtrack.
- A whole experience made of code, assets, music, mechanics, tuning, world design, and behavior.
That points toward something bigger than one demo: multi-experience generation frameworks where agents help create games, simulations, learning environments, product demos, onboarding flows, and adaptive user experiences.
The promise is compelling. Specialized agents can produce useful parts quickly, and many of those parts can be locally good.
But local success does not guarantee that the parts belong together.
That is where the real systems problem begins.
The hidden challenge was not integration.
It was coherence.
As agents generated more parts, the system needed a way to preserve direction, respect decisions already made, and decide whether each new output still belonged to the whole.
Local success. Global misalignment.
In a multi-agent system, each agent can do its job and still leave the final result incoherent.
A generated asset can be on brief and still clash with the world.
A music track can be good and still carry the wrong feeling.
A code change can pass tests and still bend the product in the wrong direction.
A research section can be accurate and still not belong to the argument.
A dashboard tile can be correct and still make the whole interface harder to understand.
The problem is not always bad output. Often, the problem is good output that does not belong.
Games make the failure loud
A game is not one artifact. It is a fusion of many domains: mechanics, code, music, visual language, environment design, story, pacing, difficulty, and player emotion.
The player does not experience those as separate parts. The player experiences one thing: one world, one feeling, one intention.
That makes games a useful stress test for a broader problem.
Other systems can hide the same failure: generated dashboards, multi-agent research reports, codebases, support workflows, onboarding flows, campaigns, and internal product experiences.
If the parts have to become one thing, local success is not enough.
The Coherence Layer
Not a box. An architectural responsibility.
The Coherence Layer becomes necessary when local task success no longer predicts global product coherence.
It does not have to live in a specific place. It could be implemented inside a lead agent, an orchestrator, middleware, a runtime, or a distributed pattern.
The location is an implementation detail. The function is the claim.
In systems where generated parts must compose into one user-facing whole, something has to own what must hold.
“The agents own how to render. The coherence function owns what must hold.”
Coherence is a property of the set
Context engineering decides what information stays available to the model. Coherence admission decides what the system is allowed to become. Different problems.
An artifact can be relevant, consistent, and high quality — and still violate a commitment that lives in no single artifact and is held by no single agent.
A visual can match the prompt and still move the world away from its accepted style. A music track can fit the scene and still change the emotional contract. A code change can pass tests and still bend the product away from its intended behavior.
A per-artifact check has nothing to catch this, because the thing being violated is the relationship between this output and everything already accepted.
Relevance is a property of one item. Coherence is a property of the set.
That is why this is not a retrieval problem, a compaction problem, or a scoring problem. It is an admission problem over writes to shared state.
Admission is delegated authority
In an agentic system, every generated artifact is also a potential write. It can change the product, the plan, the experience, the codebase, the narrative, or the operating assumptions of the system.
Coherence admission treats that write as an authority question — adjudicated against the commitments the system has already accepted, not against the artifact in isolation:
Is this change in conflict with what has already been accepted?
Should the system admit it, reject it as drift, or evict a prior commitment it now contradicts?
The question is not whether the output is good enough. It is whether it has the authority to change the whole.
This is also why admission is not scoring. A score ranks a candidate against the current state. Admission can reach back and evict a commitment the system already accepted — retracting what it once held because a new artifact reveals it as drift. Ranking never retracts.
What the Coherence Layer does
Translate
01Translate the evolving whole into per-agent constraints.
Example: A visual generator does not just need "desert." It needs Biblical, playful, readable, child-facing, not horror, consistent with the accepted world.
Evaluate
02Evaluate artifacts against the current whole.
Not only: "Is this output good?" But: "Does this belong to the set we have already accepted?"
Govern
03Decide what changes the whole.
Some outputs should be admitted. Some should be rejected as drift. Some previously accepted decisions may need to be evicted because they now pull the whole off course.
The Coherence Loop in motion
Six artifacts from a biblical-world game. Step through the loop to see how the coherence model grows — and why some outputs get rejected.
Coherence Model
Intent · constraints · accepted properties
Translator
Whole → per-agent constraints
Agentic Generators
Visual · Puzzle · Audio · Character
Evaluator
Artifact vs. current whole
Admission Control
Admit · Reject · Evict
Desert landscape — warm ochre dunes, biblical illustration style
Constraints issued
Style: biblical-illustrated · Palette: warm ochre/sand · Age-appropriate · No horror
Coherence Model — Accepted Properties
0Empty — no artifacts admitted yet
The Coherence Loop
Coherence Model
Intent · constraints · accepted properties · rejected drift · priorities
Translator
Whole → per-agent constraints
Agentic Generators
Code · Visual · Music · Mechanics · Tuning
Evaluator
Artifact vs. current whole
Admission Control
Admit · Reject · Evict
Model Update
Accepted properties shape future work
The loop is a pattern, not a prescription. Its implementation depends on your system’s shape.
The pattern is familiar from temporally validated knowledge stores: facts get written, updated, invalidated, and superseded over time. The Coherence Layer applies the same shape to commitments, constraints, style, intent, and direction — not only to facts.
Most of the loop is tractable today. Admission is the hard part.
Deciding what gets admitted into the coherence model — and what constitutes drift — requires judgment that does not reduce to a simple check.
What is buildable today? What is still hard?
Buildable today
- →Structured memory or state store
- →Translation into agent constraints
- →Per-artifact checks
- →Trace-based evaluation
- →Observability over agent decisions and tool calls
Still hard
- —Admission into the whole
- —Cross-artifact belonging
- —Eviction of drift from the model
- —Deciding what becomes part of the evolving reference
The hard part is not producing the parts. It is deciding what becomes part of the whole.
About Harel Gal
Harel Gal works on agentic AI systems, runtime alignment and reasoning, developer tools, and chaos engineering for multi-agent systems.
At AWS and Microsoft, he helped customers and companies turn AI ideas into production-ready architectures through consulting, prototypes, demos, and hands-on technical guidance.
He is the founder of Tohu.ai and is exploring the Coherence Layer as an architectural pattern for systems where autonomous agents generate parts that must come together into one coherent user-facing whole.
Where is your system making someone become the coherence layer?
If the system does not own coherence, someone downstream will.
In a prototype, that person may be the builder.
In production, it may be the user.