Genomic foundation models post strong numbers on tasks like “which organism is this coding sequence from.” The worry is that they are reading codon composition, which a lookup table could do. So I built the experiment that separates composition from arrangement.

Synonymous codon swaps let you rewrite a sequence so the protein is unchanged and the codon composition is byte-for-byte identical, while the order of codons — and the dinucleotides at codon boundaries — changes. Whatever the model still reads afterwards is arrangement.

The interest traces back to my Biosecurity Atlas, an earlier project mapping the biosafety-and-AI landscape, which got me asking how much we should actually trust these models.

• Designed a composition-exact counterfactual benchmark: a few thousand coding sequences across several taxa from a single corpus, eight rewrite conditions, and a per-sequence assertion that composition really is unchanged rather than assumed
• Scored it two independent ways — a task-coupled read-out (per-layer embeddings through one fixed linear head, balanced-accuracy drop, stratified paired bootstrap) and a label-free one (the model’s own masked-codon distribution, no head, no labels, no split) — so a result has to survive both
• Anchored everything to reference levels with no model in them: a summary-statistic classifier and gradient boosting on codon frequency
• Found order carries real but small signal, well below composition, meaning a composition-preserving rearrangement is invisible to any screen built on composition alone
• Showed a linear probe and a direct causal intervention disagree about which feature drives the decision, so decodability is not use
• Established that sensitivity to these rewrites varies widely across more than an order of magnitude in model size, with no monotone trend

Work in progress. A related manuscript was submitted to a workshop in August 2026 and is under review; nothing here is peer-reviewed. Because the application is DNA synthesis biosecurity screening, I am not publishing the specific rewrite recipes, effect sizes, or which models were defeated ahead of review. Fuller write-up on the genomic AI page.