---
title: "New models redo old work"
slug: "replay-test-for-new-models"
description: "A model launch tells you what a vendor hopes. Replaying work we have already shipped tells us what it does on our code."
date: 2026-10-16T13:26:00.000Z
updated: 2026-10-17T04:31:15.727Z
url: "https://abhirajselvaraj.com/blog/replay-test-for-new-models"
category: "Tech"
tags: ["Process", "Delivery"]
reading_minutes: 2
word_count: 332
cover_image: "https://cdn.abhirajselvaraj.com/blog/media/2026/10/c44755d1-9c62-46bc-9761-26ea2440db6b.webp"
---

# New models redo old work

A model launch tells you what a vendor hopes. **A replay test tells us what the model does on our code.** Before any new model touches client work at Evolvier, it has to redo a job we have already finished.

## What it is

We take a piece of work we shipped recently, such as a framework upgrade or a module migration. We give the new model the same starting commit and the same brief our engineers had. No hints, no cleaned-up inputs. Then we compare its output with what actually went to production.

New models are increasingly pitched for long, multi-step work like code migrations. That is exactly the kind of claim a demo cannot settle. A known answer can. We already know where the hard parts were, which tests broke and what the reviewer caught. So we know what to look for.

## What it changes

-   ✓ Decisions rest on our own code, not on benchmarks we cannot inspect.
    
-   ✓ We see where the model fails, not only where it shines.
    
-   ✓ Review is calmer, because there is a known-good answer to compare against.
    
-   ✓ Clients get a plain answer when they ask which tools we use, and why.
    
-   ✓ Adoption becomes a dated decision instead of a slow drift.
    

## How we run it

The replay runs in a sandbox branch with no client credentials and no access to live systems. One engineer owns it from start to finish and writes the result on a single page. The page covers what matched, what broke and what we would trust it with next. Short is the point.

StepOwnerWhenPick a shipped task with a known-good resultTech leadDay 1Replay it in a sandbox, same brief, no hintsEngineerDays 1-3Compare diffs, tests and review notes; decideTech lead and engineerDay 4

Sometimes the answer is yes. Often it is "yes, for this kind of task only." Occasionally it is no, and that is a useful result too.

We don't adopt models. We adopt results we have already checked.