addAI.dev·Field NotesView as Markdown

By Jay·November 2026

The Weights Set the Ceiling. Your Evals Decide How Close You Get.

#TechDeepDive#Benchmarks#AITools

On November 2, a team at MIT CSAIL published what they call "very early results" in Sparks of Weight-Space Programming. A frontier model (GPT-6 Astra) can read and write the raw weights of a neural network directly. Weights are the numbers that hold everything a model has learned, and Astra edits them with no training run at all. The authors call it Weight-Space Programming.

The headline result is a small language model (Qwen3.5-0.8B) going from 7.9% to 48.1% on the MATH-500 math benchmark after Astra edited 0.016% of its weights.

It's easy to read that as "you can now edit your way to a smarter model." The full post says something more useful for anyone who runs a business on AI. What a model learned in training sets the ceiling, and your evals (tests you run on your own work) decide how much of it you actually get.

What the researchers did

The post runs eight experiments. Three of them:

• Reading: Astra was shown the raw weights of four small networks, each trained to play a different game. It correctly named the game behind all 4 (Breakout, Pong, Pac-Man and Freeway), without being given a list of games to choose from.
• Fixing: A car-driving network, trained for 1 billion steps, failed to clear a ramp in any of 256 attempts. Astra fixed it by editing 3.5% of its weights in a single reply, using no tools. The edited network then cleared the ramp in all 1,024 attempts.
• Writing from scratch: Astra was given the rules and code of three small games and wrote a network for each in a single reply, with no training data and no tools. Each one was compared with a network trained for 500 million steps. In Breakout, Astra's network scored 517.4 against 25.3. In Asterix it nearly matched the trained one (53.2 against 55.2). In Seaquest it fell short, at 53.1 against 88.9 (about 60%).

Those are strong results on small networks, and the method is at an early stage. The experiment most relevant to anyone using language models is the math one.

The math gain was a fix

Qwen's own model card warns that this model is prone to "thinking loops": the researchers describe it as repeating itself and rarely reaching an answer. In the researchers' setup, 93% of the base model's MATH-500 answers ran to the 32,768-token limit (the cap on how long an answer can get before it is cut off).

Astra traced the looping to a single pattern inside the network and subtracted it, changing 131,072 weights. The result depended heavily on how many forward passes Astra was allowed. A forward pass is one batch run of the model on up to 512 practice problems, drawn from the GSM8K and MATH training sets.

MATH-500 and GSM8K are math word-problem sets and IFEval measures how well a model follows instructions. All three were scored on test sets Astra never saw.

The big jump only shows up at 50 passes. At 10 and 25 the loops barely went away. MATH-500 rose 4 to 6 points, GSM8K roughly doubled and IFEval more than doubled at 10. Each budget is a single run on a single small model, so treat the pattern as a lead and not a law.

This is not the model getting better at math. The ability was already there and the loop was hiding it. Most of the gain tracks answers no longer hitting the token limit, which fell from 93% to 33%.

Fixing the loops "lets the model finish answers it could already work out."

When nothing was hidden, editing did nothing

The researchers tried the same approach on Gemma 3 270M, a smaller model that rarely loops (8.5% of answers hit its limit). Astra found ordinary reasoning mistakes, such as wrong arithmetic and invalid algebra, with no single component behind them to switch off. Across 5 to 50 forward passes, MATH-500 went from 6.8% to at most 7.7%, and Astra itself reported the gain as not established.

The authors' conclusion:

Editing weights "could remove an obstacle that hid an ability, but it could not add an ability the model lacked."

That is a ceiling, at least in this test.

Reading the weights is not a substitute for testing

The post also floats a future of "auditing without evaluation": finding a model's hidden problems straight from its weights, without running tests. The experiment closest to that idea did not get there.

The team fine-tuned two copies of Gemma 3 270M, one to answer honestly and one to be sycophantic (always agreeing with the answer the user suggested). From the weights alone, Astra could not tell which was which in any of four attempts. It found plenty of differences but could not connect them to behaviour.

Then Astra was allowed to run both models on up to 16 texts it wrote, seeing only the attention patterns (where the model focused as it read) and not the outputs. With that view it picked out the sycophant in all three pairs. Running the models worked where reading the weights alone did not.

The authors' next step is an eval too. They say they are building a benchmark to turn this from "black magic into a rigorous, reproducible engineering discipline."

What this means if you run a business on AI

You are not editing model weights. You are choosing models, writing instructions and trusting outputs. Three lessons carry over:

1. The model's training is the ceiling. If a model can't do your task at all, no instruction will add the missing ability, so pick the model by testing it on your work. A launch-day benchmark score is no substitute, as we found when we looked at how GDPval scores differ between tests.
2. Sometimes the gap is something blocking the model. Qwen was stuck in a loop. Your AI tool might be stuck on a contradicting instruction, missing context or a bad format. You can't tell from the outside whether you are at the ceiling or behind a blocker, which is why you test.
3. A model has to be run to be judged. Sycophancy was invisible in the weights and showed up once the models were run. An AI drafting your quotes or replies deserves the same treatment: try it on real examples before you trust it.

A small-business eval checklist

An eval doesn't need to be fancy. Our rule of thumb (not from the paper) is 50 to 100 real examples of the task, such as past customer emails and the replies you actually sent, plus a clear definition of "right." That definition is your spec. Run every candidate model or prompt change through the same set and check it the way QA would:

• Keep a held-out set. Tune on about half the examples and judge on the rest. Astra tuned on practice problems and was scored on test sets it never saw. Tune against your test set and you optimize for the test and not the job (overfitting to the test, a close cousin of Goodhart's law).
• Check what else moved. A fix for one task can break another. The paper scored the Qwen edit on several benchmarks and you should do the same with your other tasks.
• Measure side effects such as length, cut-off answers, tone and cost. In Qwen's case the share of answers hitting the length limit moved with the gain.
• Mind the sample size. With 25 held-out examples one answer is 4 points, so run more than once and don't trust small swings.
• Make it repeatable: same examples, same settings and saved results, so you can rerun it next month. (We call this verifying from a second route, which we cover in A Change Is Not Done When the Command Succeeds.)
• Read some outputs yourself. A score can look fine while the answers don't.

Keep the change only if it wins on the held-out set and nothing else got worse.

Our view (the paper never tests this): for most businesses, a good eval is worth more than chasing a better base model.

Evals don't make a model smarter. They tell you whether you're getting what it already has.

We build the AI and the test

This is what addAI.dev does: we take one task you do every week, build the AI to handle it and build the test that shows it works on your actual work, before it reaches your customers.

Tell us your task