Evaluating the ability of language models in ranking protein variants

What does the model have to do?

Each benchmark question begins with one experimental assay. The model receives a short description of what was measured, the wild-type protein sequence, and 50 shuffled mutant sequences. It does not receive the measured fitness values, examples, an alignment, or a structure.

The model must put all 50 variants in order from highest to lowest experimental fitness. We compare that ordering with the experiment using Spearman ρ. We repeat the task across three fixed sets of variants because changing which mutations appear can change how difficult the same assay is.

For each draw, variants were ordered by experimental fitness, divided into ten equally populated bins, and sampled approximately evenly across those bins. This preserves the range of effects rather than concentrating the task near the assay median.

We benchmark fifteen general-purpose language models at their strongest evaluated reasoning setting against 95 published protein predictors across 217 ProteinGym substitution assays.

For this comparison, we take the published variant-level predictions from all 95 ProteinGym methods and recompute their scores on the same three 50-variant draws given to the LLMs. We do not use their published full-assay leaderboard scores.

Claude Opus 5 leads the language models at ρ = 0.406, beating 49 of 95 biomolecular comparators, including 41 of 46 sequence-only methods. GPT-5.6 Sol reaches 0.402, Claude Opus 4.8 reaches 0.356, Gemini 3.5 Flash 0.343, GPT-5.5 0.336, and Kimi K3 0.314.

Significant headroom still exists: the best biomolecular model scores 0.523, and the theoretical limit of the benchmark is 1.

For every model family we sweep, higher reasoning effort leads to a higher Spearman score at every step. Opus 4.8 rises from ρ = 0.209 at low reasoning to 0.356 at max.

Although a change of 0.147 can look small, it moves Opus from approximately 92nd to 74th when placed beside the 95 biomolecular models. For further context, a difference of about 0.150 separates the top specialist from the 70th place specialist in ProteinGym.

Why some rankings worked and others failed

Aggregate scores tell us overall performance, but they do not reveal the biological premise that produced it. Models can reach the right answer for a clear biological reason. They can also build a detailed and plausible argument that does not match the experiment.

The seven examples below are grouped into Success, Failure, and Improvement. Success shows a model finding the key biological premise. Failure shows that good observations do not always close the inferential loop. Improvement compares model generations on the same mutations and shows what changed in the reasoning.

Measuring benchmark contamination

ProteinGym is public, so explicit recall of a source study or measured variant could reveal benchmark leakage. Using Gemini 3.5 Flash as an LLM-as-a-judge, we classified whether a reasoning trace named a study or claimed to remember a measured result.

Recognition is common in the traces. We found evidence of it in 512 of 585 scored Sol traces (88%), 324 of 568 Opus 4.8 traces (57%), 268 of 595 GPT-5.5 traces (45%), and 61 of 651 Gemini 3.5 Flash traces (9%). Most positives identify a study or dataset. A smaller group claims to remember a measured variant result.

To adjust for assay difficulty, we analyze the 548 assay–seed cells scored by all four models. For each cell, we subtract the mean ρ of the other three models from the model's own score, then compare these residuals between recognized and unrecognized traces. We weight seeds equally and bootstrap by protein. Visible recognition does not show a consistent positive accuracy signal after this adjustment. The confidence interval includes zero for Sol, Opus 4.8, and Gemini 3.5 Flash. GPT-5.5 performs worse on recognized traces. These results support using the current splits for comparative evaluation, but they do not show that the benchmark is uncontaminated.

Some traces are more specific. Opus 5 identifies the NusA stability task as a Tsuboyama et al. 2023 dataset, while Sol names the corresponding ProteinGym CSV file when reasoning about the p53 Nutlin assay.

As frontier models trend toward full source recognition, the effect of contamination will become harder to estimate because the unrecognized comparison group will shrink. In addition, prior knowledge of a protein's structure, functional sites, or assay design may already be encoded through previous publications and related literature.

The audit therefore suggests that the current splits remain useful, while also showing why this field will eventually need a private evaluation built from unpublished assays.

How does PG-LLM perform on newly released data?

To reduce potential contamination, we evaluated models on 59 assays from 19 source studies whose scores became public after January 2026. We used the same N = 50, no-tools protocol across three fixed candidate draws.

Additional test-time compute improves all three model families. GPT-5.6 Sol rises from ρ = 0.271 at low effort to 0.332 at max, approximately matching VespaG at 0.323. Claude Opus 4.8 and Gemini 3.5 Flash show similar scaling trends. The persistence of these trends on recently released assays suggests that PG-LLM captures generalizable protein reasoning and is unlikely to be explained solely by prior exposure to benchmark scores.

Interactive flow diagram. Use Tab to move through the search, date-filter, and final-set steps; activate a step to keep its details open.

Interactive effort-scaling chart. Use Tab to inspect every model-effort point and the two biomolecular reference lines; activate a point to keep its details open.