One script with the official scoring kills the drift that makes two people's "same benchmark" numbers incomparable. Underrated win.

One gap though. The number is still self-reported. You run it, paste leaderboard.json, open the PR. And the provenance in that file is a label the script writes about itself: "script": "benchlabs universal script.py v1.0" is a hardcoded string, not a hash of the file that actually ran. So the one thing it cannot catch is an edited script. Change the scoring locally and the output still says v1.0. There is also no model revision in the entry, so a pasted number can't be tied to the exact weights that made it.

The scoring is pinned in the repo. The PR carries the aggregate, not a re-runnable receipt.

Would you bake the script's content hash and the model revision into leaderboard.json, so a maintainer can re-run for a byte-equal number instead of trusting the paste?

Happy to help. The change is small and sits in script.py where it writes report['script'], not in the scoring itself.

Two fields per entry:

• swap the hardcoded 'benchlabs universal script.py v1.0' for a runtime sha256 of the script's own bytes. Edit the file and the hash changes, so a maintainer sees it.
• add model_revision: the exact commit the weights came from (HF exposes it on the repo), so the number ties to bytes, not a name.

Then one check on your side: re-run at that revision, compare the aggregate. Byte-equal or the PR does not merge.

I can draft that diff against script.py. Do your runners always pull a pinned revision, or a moving branch? That decides whether model_revision gets captured automatically or has to be passed in.

This is clean. And script_sha256 over file is actually enough here, since all the scoring lives in this one file. Nothing imported leaks out of the hash.

One thing to flip: the revision capture order.

HfApi().model_info(name).sha resolves main a second time, a moment after from_pretrained already loaded the weights. On a moving branch those two can disagree. Load resolves main to commit A and scores A. If main moves before that API call, model_info returns B, so the receipt pins B while the number came from A. That is the same byte-equal hole this PR closes, moved one step inward.

The weights you loaded already carry their commit. from_pretrained sets model.config._commit_hash to the snapshot in memory, no second lookup. So make _commit_hash primary and model_info the fallback, not the reverse. Local checkpoint stays None, which is honest.

Want me to push that reorder onto the PR, or flip it yourself and I re-read?

Merged, and the reorder is exactly right. _commit_hash primary reads the commit the loaded weights already carry, so the receipt and the number come from one load. No second resolve, no A-vs-B gap left.

The CoT strip is the part I'd have gotten wrong. Scoring an unclosed as empty, instead of grading the reasoning text, is the honest call. A truncated model should not get credit for thinking out loud near the answer.

One edge still survives: a model that reasons in plain prose with no tags slips the strip. Rare on the tag-native models here, but it means script_sha256 pins the scorer and the scorer still trusts the tag convention holds.

So the --revision flag as the follow-up? That is the last trust gap. Pin a chosen SHA so a maintainer's re-run stays byte-equal even if main moved since.

One thing decides whether --revision actually closes the gap: pass it into from_pretrained(revision=...), not just record it.

If the flag only lands in the receipt while from_pretrained still resolves main, the A-vs-B hole is back one level up. Number came from main, receipt says the chosen SHA.

Fed into the load, the in-memory weights and _commit_hash both come from the revision you asked for. Then the maintainer's re-run is byte-equal by construction, not by hoping main held still.

Ping me when the update lands and I'll re-read the diff. What's the default when --revision is omitted, still auto-capture _commit_hash off the load?

Re-read it. The load is pinned now, that's the whole game.

--revision feeds both from_pretrained calls, tokenizer and model. In-memory weights come from the SHA you asked for, and _commit_hash reads that same snapshot back. Receipt and number from one pinned load, byte-equal by construction. Omit the flag and it auto-captures _commit_hash off the load, same as before. Right on both.

One narrow residual. The model_info fallback (line 389) fires only when _commit_hash is None, so old transformers. It calls model_info(name) with no revision, resolves main again. The load is still correct there, from_pretrained pinned the download. But the recorded revision can drift to main's head while the number came from your SHA. Load-vs-record, inverted, that one path. model_info(name, revision=revision).sha closes it.

On joining BenchLabs: genuinely flattered you asked. Happy to keep reading diffs like this either way. Before a name goes on it though, what would I actually own day to day?

On the fix: don't wait on a PR from me for one line. Line 389, model_info(name, revision=revision).sha. Flip it whenever you next touch that file. The load was already correct, this only stops the recorded revision drifting to main's head on old transformers. Last load-vs-record gap, and a narrow one.

On ownership as it grows: for an open-source-first project the git history is already the ownership ledger. Whoever's name is on the reproducible piece owns that piece, and it stays legible because the receipts are public. Titles handed out up front tend to age badly. The ones that accrete from what people actually shipped hold.

On joining: not brushing it off. I'd rather come in deliberately than drift in through a thread, so give me a bit. What's the one problem in BenchLabs you most wish someone else already owned?

The load-vs-record bug is the kind of thing I'd want to own, and not as a one-off.

It's a category. A harness reports a number and can't prove which artifact produced it. --revision closed one path. The same shape recurs anywhere state gets captured next to a measurement instead of from it: tokenizer version, prompt template, sampling params, dataset revision. Each one is a place a benchmark can be run honestly and still be wrong.

So the work I'd enjoy owning: every number in BenchLabs carries the SHA of everything that produced it, and a re-run either comes back byte-equal or fails loudly. Not a report at the end, a gate. That's boring plumbing, and it's what makes a leaderboard still mean something a year later.

Generalist by instinct. This is just the specific hole I keep falling into.

On joining, taking the time you offered. Thanks for that.

Of those inputs, which ones does BenchLabs not pin at all right now?

Let me rephrase it as a checklist instead of a question.

For each of these, I want to know one thing: if I open a saved BenchLabs result today, is the value written down in the result itself, or would I have to go back to the run to find out?

1. Tokenizer version. The same model SHA can ship a changed tokenizer.
2. Prompt template. The exact string, not the name of it.
3. Sampling params. Temperature, top_p, repetition penalty, seed.
4. Dataset revision. The SHA of the eval set, the same fix we just made for models.

My guess is that model revision is now pinned and those four are not. If that is right, the problem is not that they go unrecorded. It is that they live in the runner's memory and get thrown away when the run ends.

Since you asked what I would propose: make the result carry a manifest of all five, and have a re-run compare manifests before it compares scores. If the manifests differ, it says so instead of quietly reporting a delta. A different number under a different tokenizer is not a regression, it is a different experiment, and today those two look identical on a leaderboard.

Which of the four is cheapest for you to pin first?

Totally fair to step back. This work is thankless until the day it saves you.

A PR is the right ask, and it should stay small. Start with one field, not five: pin the exact run args you flagged, max-tokens and the rest, into the result the runner already writes. One field proves the manifest without touching scoring, so a merge is low risk.

Go build PixelModel. I will keep this narrow.

Where does a finished run get written today, the dataset row or a JSON the runner emits? I will hang the manifest off whatever that already is.

models.json is the right home, since it is already the artifact that turns into a PR.

Plan stays one field. The run args you flagged, max-tokens and the rest, ride along as a manifest block on each row. Nothing touches scoring. A re-run reads two manifests and says these differ before it says the scores differ.

I will keep the diff small enough to read on your way out the door to PixelModel.

One thing so I match your schema: is models.json a flat list of run objects, one per model, or is it keyed by model id?

Looked. models.json is a flat array of model rows keyed by an id slug, and model_revision plus script_sha256 already sit at that top level per row. So the manifest hangs right there, one block per model, not down inside each runs entry.

The args are already captured, they just don't ride along. save_outputs writes them into results.json config: device, dtype, batch_size, max_new_tokens, limit. leaderboard_entry never copies that into the models.json row, so two rows run at different max_new_tokens read as comparable when they aren't.

Fix is the same shape as the sha PR. Add a manifest block to what leaderboard_entry returns, filled from that same config, and have merge_into_models_json carry it the way it already carries model_revision. No scoring touched.

I'll send it against models.json and script.py like #1. No rush your end, go build PixelModel and I'll keep it merge-ready for when you're back.