Introduction⌗

You ask an AI agent to rewrite an old service in a new language. The code it writes looks clean, the tests it wrote for itself are green, and it tells you the endpoint is done. Then you send the same request to the old service and to the new one, and the two answers aren’t the same.

A few months ago we had to move a working prototype into a production codebase, in another language and another framework, and we wanted Claude Code to write most of it. I didn’t trust “done” from the agent, so before the first endpoint was rewritten we built a small check around it. This post is about that check, how the work ran around it, and what it still misses.

The old system already has the answers⌗

The hard part of a rewrite is that nobody wrote down what the old code does. The code is the only description, with every odd case somebody added on a Friday.

So we didn’t try to describe the behaviour to the agent. We asked the old system. For each endpoint we made a list of requests: the normal ones, the empty ones, wrong input, and the edge cases we found by reading the handler. We ran them against the old system and saved every response to a file. Those files were then frozen, and from that point they were the definition of correct.

None of this is new. It is called characterization testing, or golden master testing, and people used it for rewrites long before AI. What changed is who does the typing in between.

One endpoint at a time⌗

The work ran in the same order for every endpoint:

  1. The agent reads the old handler and lists the cases worth asking.
  2. It runs those cases against the old system and freezes the answers.
  3. It writes the new code from the frozen answers.
  4. A script starts the new code, sends the same requests, and compares each answer with the frozen one. If one answer is different the script exits with an error, and the endpoint is not done.
  5. We commit, write down what went wrong in this round, and take the next endpoint.

Each step ran in a fresh session, with the context cleared in between. That was first meant to keep the agent from mixing up what the old code does with what it had just written, and it also turned out to be cheaper: on the two endpoints where I measured it, the tokens dropped by about three quarters.

Why a script and not the agent⌗

The agent that wrote the code is a poor judge of whether the code is right. It wants to be finished, and it will tell you a small difference doesn’t matter. A script has no opinion. It compares two files and returns zero or it doesn’t, and the agent can’t talk its way past an exit code.

The differences it caught were mostly small and boring. An amount that the old system sent as text and the new one sent as a number. A field the old system sent as null and the new one left out. A list that came back in another order, a date in another format. The agent had called every one of those endpoints done, and each of them would’ve broken a screen somewhere.

We saw the same thing with the instructions around the work. Rules we wrote down for the agent were followed for a while and then skipped, until we turned them into a check that stops the work. I wrote about that pattern in the post on review hooks, and it held here as well.

What it doesn’t catch⌗

The check is only as good as the questions, and it fooled us once. Early on the board showed every endpoint green, while the endpoints shared test data and that shared data was hiding failures. After that each endpoint got its own data.

Some fields change on every run, like generated ids and timestamps, so they have to be masked before the comparison, and what you mask you don’t check. Almost every case had at least one masked field. File downloads and other binary answers weren’t compared at all.

Login was left out on purpose, because the new system handles it in a different way and the answers were never going to match.

The old system’s bugs come along too. If the prototype rounds a total the wrong way, the new code has to round it the wrong way as well, or the script fails it. Same answers means the rewrite is faithful, and that is all it means. It says nothing about security either, so that testing came afterwards and found things the comparison never could.

Conclusion⌗

The rewrite took about four weeks, one endpoint after the other. When the last endpoint was through, we threw the scripts and the frozen answers away, because they were built for this one move and nothing else.