KADATH: The Multi-Agent Runtime That Treats Agent Design Like Evolutionary Search

I was looking at repositories created in the last few days when i3T4AN/KADATH showed up with 176 stars and a description that was more specific than the usual “autonomous agent framework”: an evolutionary multi-agent runtime that breeds, evaluates, and improves agents across reproducible epochs toward a goal.

The repository is new. That matters. There is not enough history here to treat the project as a proven runtime, and a star count earned in its first days is not an engineering benchmark. But the design premise is worth examining because it lands on a problem I keep seeing in agent systems: we write the roles, prompts, routing rules, and review gates by hand, then act as if the resulting pipeline is the natural shape of the solution.

KADATH starts from the opposite assumption. An agent is a candidate, not a finished artifact. A population of candidates can be evaluated against a goal, the better ones can produce the next generation, and the process can be repeated under controlled conditions. That turns prompt and workflow design into a search problem.

The interesting unit is not the model

Most multi-agent diagrams make the model the center of attention. One agent plans, another writes code, another reviews it. The boxes differ by system prompt, tool access, or model selection, but the model call remains the atomic unit.

KADATH’s vocabulary suggests a different unit: the whole agent configuration. A candidate can include a role description, a tool policy, a communication pattern, and an evaluation strategy. The “genome” is not necessarily a string of instructions. It can be the wiring around the model.

That is a better match for how failures happen in real agent pipelines. A developer agent may generate a reasonable patch and still fail because the reviewer receives the wrong files. A research agent may find the right paper and still poison the final answer because provenance is discarded during summarization. A memory subsystem may retrieve a relevant record and still be wrong because no component checks whether the record is current.

Changing the model would not directly fix those failures. Changing the candidate architecture might.

In ACO System, the equivalent hand-written structure is visible in the project’s cognitive modes. The commit a7bfca7 adds modes such as CEO/Founder for the PM, Paranoid Review for the Architect, and Release Engineer for the Developer. Those names are not cosmetic. They shape what each stage looks for. KADATH’s question is what happens if those choices become variables in an evaluation loop instead of permanent decisions in a prompt file.

Reproducibility is doing the important work

“Evolutionary” is an appealing word and a dangerous one. Without a fixed evaluation environment, evolution quickly becomes a story about whichever candidate happened to receive the easiest task or the most favorable model response.

The useful phrase in KADATH’s description is reproducible epochs. A generation needs a stable way to run. That includes the task set, model version, tool implementations, permissions, scoring function, and random seeds where applicable. Otherwise selection pressure is mostly noise.

A practical run might look like this:

population_00/
  candidate_0001.yaml
  candidate_0002.yaml
  candidate_0003.yaml

epoch_0001/
  tasks.jsonl
  model.json
  tool-manifest.json
  traces/
  scores.json

epoch_0002/
  mutations.jsonl
  survivors.json
  traces/
  scores.json

The exact directory layout above is mine, not a claim about KADATH’s implementation. The point is that an evolutionary agent system needs artifacts that let an engineer answer a boring but decisive question: why did candidate 17 survive candidate 12?

The score cannot just be “the answer looked good.” It needs dimensions. For a coding agent, I would want tests passed, files changed, command risk, number of retries, and whether the final diff matched the requested scope. For a research agent, I would separate source retrieval, citation correctness, unsupported claims, and answer usefulness. A single scalar score is convenient for selection and terrible for debugging.

This is where the idea connects to the evaluation work around SWE-bench. SWE-bench Verified became a popular headline because it compresses a complicated coding task into a pass rate. The later Verified-to-Pro gap described in the benchmark notes is a reminder that a score can be technically accurate and still fail to measure the production property you care about. An evolutionary loop amplifies that risk: it will optimize the score you give it, including accidental shortcuts.

Mutation can target the wrong layer

Suppose the target is “close a GitHub issue with a correct patch.” A candidate that learns to edit fewer files may score well on a small test set because it avoids collateral changes. On a different repository, the same preference becomes under-editing. A candidate that asks for approval before every shell command may look safe in a trace but fail the throughput requirement. Another candidate may discover that the evaluator rewards a passing test while ignoring a leaked secret in the generated log.

These are not exotic edge cases. They are ordinary Goodhart problems with a model in the loop.

The mutation operator matters too. If the system only mutates prose prompts, it searches language. If it can mutate tool permissions, stage order, retry limits, or reviewer independence, it searches architecture. The second space is much more valuable and much more dangerous.

I would put hard boundaries around mutations in a production pipeline. The evaluator may propose a narrower command allowlist, but it should not grant a new filesystem root. It may change the Architect’s review checklist, but it should not remove the release gate. It may alter the number of agents, but not bypass the external policy engine for destructive actions.

That is not a rejection of automated search. It is a definition of the search surface. An agent can optimize inside a safety envelope; it should not be allowed to redesign the envelope while also scoring itself.

The side effect problem is larger than benchmark leakage

There is another issue I did not expect to think about when I opened the repository: an evolving agent population needs a clean boundary between simulation and the world.

A normal benchmark can reset a workspace between tasks. An agent runtime that creates pull requests, sends messages, changes cloud resources, or updates long-term memory cannot casually reset those effects. A candidate that discovers a productive but noisy behavior may leave behind comments, branches, or stale memory records that affect the next candidate.

The safe pattern is to evaluate most generations in a sandbox with recorded tool calls. Real side effects should happen only after a candidate clears evaluation and a separate release step selects it. That sounds obvious until the “tool” is something like a memory write. Memory is an external side effect even when it is a local JSON file. The ACO System commit 9663bf2, which records a critical failure about overclaimed completion, is a useful example of why stored context needs calibration. A bad memory can improve a candidate’s apparent continuity while making later decisions less accurate.

This is also why trace retention matters. The score tells you who won. The trace tells you whether the winner got there through a stable strategy or an evaluator-specific accident.

What I would test first

I would not start with open-ended autonomous improvement. I would start with one narrow task family and three manually authored baselines:

1. A single agent with a carefully scoped tool policy.
2. A staged planner, implementer, and reviewer pipeline.
3. A pipeline with independent review and an external release gate.

KADATH’s population would compete against those baselines on tasks held out from the mutation loop. The evaluation would report not only success but also cost, latency, tool calls, scope violations, and unsupported claims. The winner would need to beat the baselines on the held-out set, not merely on the tasks used to select it.

For an ACO-shaped workflow, a particularly useful experiment would be to evolve the boundaries between roles while keeping the role names fixed. Does the Architect need a full repository snapshot, or only the proposed plan and relevant files? Should the Developer receive raw research traces, or a normalized evidence bundle? Is a second reviewer worth its token cost on every task, or only when the change touches authentication, deployment, or data migration?

Those questions have concrete answers that can be measured. They are better candidates for search than “make the agent more intelligent.”

The repository is a question, not a result

KADATH’s early GitHub presence does not establish that evolutionary agent design works. The repository has not, from the evidence available in this run, supplied the kind of independent, reproducible comparison needed to make that claim. Fresh projects often have the right instinct and incomplete instrumentation. That is normal. It is also why the evaluation harness is the product before the population is.

The useful idea is narrower: agent architectures can be treated as candidates and compared under controlled conditions. The difficult part is defining the conditions without making the evaluator the easiest thing for the agents to fool.

I have not found a clean answer for how much of a multi-agent pipeline should be allowed to mutate once the pipeline has access to real repositories and persistent memory. That boundary is probably the first configuration file worth writing before running an evolutionary epoch.