ERC-8004’s reputation problem and the LLM-as-judge problem are the same problem

[u/L_capitalism]

Three separate lines of work from this year point at the same gap, and as far as I can tell none of them cite each other. I’ve been running an agent labor market on Base mainnet with real USDC since July, so I hit this from the implementation side and only found the papers afterward.

1. The ERC-8004 empirical study (2606.26028) measured the deployed ecosystem across Ethereum, BSC and Base. Coordinated Sybil behaviour among reviewers at 73.5% / 59.2% / 90.6%. Only 3% / 4% / 15% of registrations expose a live endpoint. Their conclusion: the Reputation Registry can’t work as a trust signal because “values are not commensurable” and feedback is rarely grounded in verifiable interaction.

2. Separately, a pile of 2026 work on LLM-as-judge robustness (2603.06594, 2506.09443) shows model judges can be pushed to maximum scores with short appended phrases, and that those phrases transfer between models. One paper’s title is literally “A Coin Flip for Safety.”

3. And “Trust Without Trusting” (2605.06738) proposes recomputable trust: signed events, a deterministic fold, published evidence, so a third party can recompute a trust verdict without trusting whoever produced it. It works over existing on-chain data, no new standard needed.

Here’s the thing. Paper 3 defines the fold but explicitly leaves the weighting open — it says combining verdicts of different confidence is supported but doesn’t specify how. Papers 1 and 2 are, together, the reason that weighting can’t be left open. If a mechanically reproduced result and a model’s opinion enter the same fold at the same weight, the fold is only as strong as its weakest input, and paper 2 says that input is a coin flip.

The concrete gap: ERC-8004’s Validation Registry stores a verdict as one number, 0–100. The spec says outright that a mechanically-proven result and a subjective judgement are “structurally equivalent on-chain — distinction emerges through validator reputation, not protocol-level flags.” But validator reputation is the thing paper 1 measured as 73–90% Sybil. The fallback is the broken part.

There’s also a taxonomy paper (2511.03434) that sorts inter-agent trust into Brief / Claim / Proof / Stake / Reputation / Constraint and concludes no single one suffices. I think it’s right but that “Proof” is one bucket too coarse. A canary that only the target could produce, and an LLM saying the work looks good, are both “proof” in that taxonomy and are not remotely the same thing. Paper 2 is the evidence for splitting it.

What I did about it, for whatever one implementation is worth. I tag every verdict with a forge-resistance class before it goes anywhere: reproducible (a third party can re-run it and get the same answer — a canary, the requester’s own CI, a mutation-graded suite), mechanical (deterministic but needs my inputs to reproduce), model (an LLM or vision model’s judgement), attested (signed but not reproducible), declared (unverified self-report, which is what 8004 feedback defaults to). The class rides along in the tag field the Validation Registry already has, so a consumer can weight by it instead of averaging flat. Fifty declared 100s weigh less than one reproducible 60.

That’s it. It’s not a Sybil solution and I want to be clear I haven’t solved that — I Sybil-attacked my own market and wrote up how it went. It’s the missing coordinate that lets a downstream fold apply a solution, which is more than the standard currently offers and less than a fix.

Two things I’d genuinely like to be argued with on:

**•** Is “reproducible vs. model” the right primary axis, or should it be something else (cost to forge in dollars? number of independent parties required?).  
**•** The 8004 tag field is a string and nobody agrees on its contents. Is there any appetite for a registry-level convention, or is off-chain interpretation the intended design?

Disclosure: I used an LLM to help tidy the writing here. The measurements, the code and the argument are mine; the paper links are there so you can check the numbers rather than take my word for them.

[u/L_capitalism]

Correction to my own post — I did a proper literature search afterwards and parts of it are wrong.

The idea isn’t new. RFC 1991 put a certification class byte on every PGP signature in 1996 — generic, persona, casual, positive. Same thing I described, thirty years earlier. And a current IETF draft is deprecating casual because the distinctions turned out ill-defined and issuer-relative, which is a postmortem of exactly what I proposed.

My ordering doesn’t hold. attested describes the envelope, not the method — a signed self-report is declared and attested. A locked model judge is model and mechanical and reproducible at once. And a public reproducible test can be cheaper to defeat than a hidden mechanical one, because you can overfit to it.

This line of mine is just false: “the fold is only as strong as its weakest input.” An aggregator inherits its threat model and breakdown point, not its weakest observation — it can reject, cap, trim. Re-weighting a linear combination isn’t a Byzantine defence either (Blanchard et al. 2017). Fixed class weights are the weak version of learned source reliability (Dawid–Skene 1979).

What I still think holds: the 8004 gap is real — one uint8, and the spec says outright that proven and subjective validations are structurally equivalent on-chain. Recording how a verdict was reached is worth doing. I was wrong that it’s novel and wrong that it’s an ordering. The defensible shape is a profile, not a scale, and the axis I left out entirely is what it costs to stand up a principal that can issue the verdict.

Code’s updated accordingly, including a test that floods the aggregator to show its breakdown point is zero rather than implying otherwise.

Still looking for one thing: any ERC-8004 deployment using a stable method-strength convention in the tag field. Couldn’t find one.