The Audit Asymmetry Problem
Capacity conditions for substrate-independent audit
1 Introduction
The first note in this series identified a class of evaluation failure — category-level failure — that distribution metrics do not capture, and proposed three structural conditions an audit instrument would need to satisfy in order to block such failure: provenance independence, external anchoring, and speed parity (Ortulanu, 2026). The note observed that current substrate-independent audit work, conducted by frontier safety institutes and similar bodies, satisfies these conditions only in part, and closed by flagging an open question. The argument that self-referential evaluation operates within a signal-to-noise regime whose floor cannot be raised by within-substrate refinement (Ortulanu, 2026 §7.2) was framed as an analogy to channel capacity rather than a formal result; the section on limitations (Ortulanu, 2026 §8) identified the missing piece explicitly: “A formal treatment would require an information-theoretic model of substrate-shared evaluation that goes beyond the present argument.” A separate open question (Ortulanu, 2026 §9, Substrate Ancestry) asked at what distributional threshold audit independence is restored.
The present note takes up that open question. It argues that the conditions under which substrate-independent audit ceases to be informative can be specified along three dimensions of an audit relation’s capacity — bandwidth, semantic reach, and self-reference depth — held relative to the requirements that the audit task induces from the auditee. When the auditor’s capacity falls below the task-relative requirement on at least one dimension, and no compensating institutional condition restores the missing dimension, the audit cycle’s judgments cease to track the task-relevant state of the auditee, regardless of the auditor’s incentives, access, or authority.
The claim here is narrower than structural impossibility. The note maps the conditions under which audit ceases to be informative, and traces those conditions to a vector of structural limits — each named through a formal source (Shannon, Rice, Gödel) whose role in the present argument is to mark the shape of a constraint rather than supply its derivation.
The framework decomposes problems often grouped under the common heading of complexity into three dimensions whose deficits produce different failure profiles and admit different forms of approximation. It locates audit failure in the capacity relation between auditor and auditee, separating capacity conditions from the institutional conditions of incentives, access, and authority — where the existing literature is concentrated.
Where the first note describes category-level failure on the auditee side, the present note describes the auditor-side relation that produces it. Category-level failure is one surface form that capacity asymmetry takes when task-relevant distinctions are merged, delayed, or displaced by an audit relation whose capacity no longer meets the task it is asked to perform.
2 The Formal Structure of an Audit Relation
2.1 Audit cycle and task-relative capacity
An audit cycle consists of an auditor A selecting or receiving observations from an auditee S, through outputs, samples, disclosures, queries, or tests, and producing judgments in a description language L_A for a stated audit task τ. The audit task τ specifies which features of S the judgment is meant to track; what falls outside τ’s scope is irrelevant to the present analysis. The capacity required by τ is therefore not a property of S in full, but a property of S restricted to what τ asks about. Write R_τ(S) for the task-relative capacity requirement: the rate, expressive class, and representational depth that the audit task τ induces from S.
2.2 Three dimensions
Three dimensions of A’s capacity, each held relative to R_τ(S), condition the audit. Each is named through a formal source whose role is taken up at length in the following sections.
Bandwidth. The rate at which A records the task-relevant stream from S without loss that changes the audit judgment. The constraint is information-theoretic, in the sense of Shannon (1948); section 4 develops it.
Semantic reach. The class of task-relevant properties of S that A’s description language and decision procedure can distinguish. The constraint is computability-theoretic, adjacent to Rice (1953); section 5 develops it.
Self-reference depth. The number of representational levels at which A describes S’s representation of itself, and of A, without allowing the audit criterion to collapse into the object being audited. The constraint is consistency-theoretic in shape, after Gödel (1931); section 3 develops the audit analogue.
2.3 Capacity asymmetry
A’s capacity is a triple (b, e, d) compared component-wise against R_τ(S). Capacity asymmetry obtains when A falls below R_τ(S) on at least one relevant dimension, and no compensating institutional condition restores the missing dimension. Under such asymmetry, the audit cycle ceases to be informative: A’s judgments stop tracking the task-relevant state of S — too coarse for what τ asks to be distinguished, or too late for the state on which third parties act. Institutional repair — independence, resourcing, transparency — leaves informativeness untouched when the relevant capacity deficit remains in place. The outputs may retain the form of audit judgments — scores, rankings, certifications — while no longer tracking the task-relevant state of the system.
Loss of informativeness sits below ordinary error. An audit may err while remaining informative — its judgments may be wrong while still tracking the categories the task requires it to keep separate. Loss of informativeness is the systematic merger of distinct task-relevant states into a single judgment, or the arrival of judgments at a latency that makes the tracked state no longer the state on which third parties act. The threshold is therefore task-relative: the same audit cycle may remain informative for one task and lose informativeness for a structurally adjacent one, when the distinctions the second task requires lie outside what the audit can record, distinguish, or stand outside.
2.4 Non-orthogonality and cascade
The interdependency among the three dimensions is structural before it is temporal. Bandwidth determines what enters the audit record. Within that record, semantic reach fixes which distinctions can be drawn. Self-reference depth then concerns whether the criterion used to draw those distinctions stands outside the object whose behavior it judges. Each dimension’s exercise depends on what the previous dimension supplies. Bandwidth deficit forces compression, and so narrows the subclass of S-behavior over which L_A can distinguish; semantic reach deficit flattens the representational depth at which A can model S’s self-description; self-reference depth deficit, when A and S share substrate, returns A’s judgments to a self-evaluation that narrows semantic reach again. The condition the note names — capacity asymmetry — is therefore a static interdependency among three constraints, not a sum of three independent thresholds.
The relation among the dimensions is not only cascading. Maximizing one dimension can diminish another: an auditor whose substrate is fully disjoint from the auditee’s may meet self-reference depth at the cost of the semantic reach that comes from shared representational primitives. The audit task therefore does not call for maximizing each dimension independently. It calls for the least separation from the auditee sufficient to meet R_τ(S) on each dimension simultaneously — close enough to read what the auditee produces, distant enough that the criterion forms outside what is being judged. In repeated audit cycles this structure can appear as cascade; the dynamics of that appearance are treated in subsequent notes in this series.
2.5 Scope and differential diagnosis
The note separates capacity conditions from institutional conditions. Access, incentives, and authority determine whether audit is permitted, motivated, or enforceable. Capacity determines whether, even under favorable institutional conditions, the audit judgment can remain informative. Where the existing literature on audit failure addresses incentive misalignment (issuer-pays arrangements, regulatory-license capture, reputational equilibria), the present analysis addresses what fails when incentives are aligned. The two analyses are complementary; this one operates on a different layer.
The capacity account is distinguishable from incentive and organizational accounts by its counterfactual. It predicts audit failure even under aligned incentives, single-agency certification, and integrated organizational structure, whenever R_τ(S) exceeds A’s capacity along a relevant dimension. Where institutional conditions are poor but A’s capacity meets R_τ(S), the account predicts corruption or capture, not the specific category-level loss of informativeness analyzed here. The vector names only the capacities required for judgment formation under favorable institutional conditions: recording, distinguishing, and standing outside the recursive formation of the criterion. Access, incentives, authority, and strategic adaptation by S enter the analysis when they alter one of these capacities or the requirement R_τ(S).
When R_τ(S) cannot be met by A’s capacity, the institutional response is rarely to acknowledge the capacity deficit and revise the audit instrument. It is more often to narrow τ — to redefine the audit task until it lies within what the existing capacity can meet. Call this task contraction. The audit then succeeds in form: it produces judgments that satisfy its own redefined criterion. What it audits is no longer what the original task asked it to audit. The contraction need not be intended as concealment; the framework is silent on the actor’s awareness. Whether or not participants recognize the deficit, institutional pressure to produce judgments selects for a τ the existing capacity can meet. Task contraction is one form of the same failure — the institutional shape capacity asymmetry takes when the asymmetry is denied or unrecognized.
3 The Gödel Constraint — Self-Reference Depth and the Substrate of Audit
Self-reference depth has two components. The first is recursive modeling depth: how many levels of S’s self-description A can represent without the representation collapsing into the object it represents. The second is substrate separation: whether the criterion used at those levels is formed outside the substrate whose behavior the criterion judges. The present note is concerned chiefly with cases in which the audit task requires both. Gödel (1931) names the formal shape of the constraint that makes the second component necessary in sufficiently expressive settings: in such settings, consistency cannot be certified from within the same system. The audit analogue arises when A’s description of S, S’s self-description, and the criteria of consistency occupy the same representational substrate.
The formal theorem and the audit analogue must be held apart. Gödel’s result is conditional on the formal expressivity of the system in question and is silent about the wider class of representational substrates that audit relations can occupy. The audit analogue is not derived from the theorem. The analogue names the shape of an audit relation in which the depth at which A can model S’s self-description, without paradox, is bounded by the substrate the two share. The substrate-sharing constraint is broader than the consistency-certification limit Gödel proves, and weaker as a result. It does not produce a paradox. It marks the point at which the standpoint τ requires cannot be supplied from inside the substrate that produced what is being judged. When that shape obtains, A’s judgments about S’s consistency become judgments about a system A is not external to; the criterion of consistency is co-produced with the artifact whose consistency is at issue.
Self-reference depth deficit concerns the audit component that requires the auditor to stand outside the audited substrate. Other components of audit — observational throughput, distinguishing among task-relevant properties — proceed within their own constraints. Where R_{τ,d}(S) requires depth that the shared substrate does not permit, that component of the audit task ceases to be informative; the remaining components continue to operate.
Once separated from the theorem, the analogue does different work: it opens an empirical question rather than adding a further theoretical limit. The static condition named here — that A and S share substrate at the moment of judgment — admits a diagnostic test. The substrate is shared to the degree that A’s representational primitives, training history, and decision procedure are drawn from the same sources as S’s. Where the sources are disjoint, the depth condition can be met from outside; where the sources overlap fully, no internal procedure stands in for the external standpoint R_{τ,d}(S) requires. Most actual audit instances lie between these endpoints. The question for a given audit task is then how far along that scale A and S sit, and whether that distance suffices for τ. In the contemporary case where systems in a class share substantial training-data lineage, the relevant question shifts: not whether substrate separation is met, but along which dimensions A’s lineage is structured to differ from S’s, and whether that structured difference is what τ requires. How shared substrate grows or diminishes when audit cycles repeat — when the systems trained on an audit’s outputs become, in subsequent cycles, the systems on which the audit operates — is taken up in subsequent work in this series.
4 The Shannon Constraint — Bandwidth and the Compression That Audit Forces
The bandwidth dimension is the rate at which A can record the task-relevant stream from S without loss that changes the audit judgment. Shannon (1948) names the formal constraint: a channel has a capacity beyond which faithful transmission cannot be sustained, and beyond which lossy compression becomes a necessity. The audit consequence the present note adds is that when compression is forced, the structure of that compression determines which categories of S-behavior survive into A’s record.
In audit relations, rate has both a throughput and a latency face. When the task-relevant state of S changes before A’s judgment can be formed and acted upon, the audit relation is bandwidth-deficient for that task even where individual observations are locally accurate. Speed parity in the first note is the latency side of the same constraint — the requirement that the state judged remain the state on which third parties act.
The compression that bandwidth deficit forces is selective rather than uniform; the selection is governed by what L_A is equipped to record. Categories of S-behavior that map cleanly onto L_A’s primitives survive into the audit judgment; categories that require expressive resources L_A does not possess do not survive at all, regardless of whether they continue in S. A failure mode for which L_A has no available category does not enter the audit record as a failure mode; it enters, if at all, as noise or residue. This is the auditor-side mechanism by which an audit cycle that begins under bandwidth pressure produces judgments that are systematically blind to behaviors of a particular shape. The first note in this series (Ortulanu, 2026 §7.2) observed an analogous regime through an information-theoretic analogy — a signal-to-noise floor in self-referential evaluation that refining within the same substrate cannot raise. Ortulanu (2026 §8) flagged the analogy as informative but not formal, and invited an information-theoretic model of substrate-shared evaluation that the analogy itself does not supply. The present section provides one component of that model: the relation between channel capacity, forced compression, and category survival.
Bandwidth requirements are also task-relative. The audit task τ specifies which features of S’s output stream the judgment is meant to track; what falls outside τ’s scope does not contribute to R_{τ,b}(S). An audit that is bandwidth-adequate for one task may be bandwidth-deficient for another task on the same auditee, because the task-relevant stream is denser. This task-relativity is what permits the same auditor — say, a credit rating agency — to render informative judgments about one class of instruments while rendering uninformative judgments about a structurally adjacent class: not because the agency’s institutional condition differs across the two tasks, but because R_{τ,b}(S) for the second task exceeds available bandwidth, and the available bandwidth is the same in both cases. The 2008 forensic instance taken up in section 6 is one case where the bandwidth requirement of the structured-finance audit task rose faster than the auditor’s observational apparatus could match.
The constraint has consequences for substitutability. Bandwidth, unlike provenance independence and external anchoring, cannot be approximated by procedural repair alone. A channel whose capacity falls below the rate of the signal it is meant to record can be re-staffed, re-resourced, and re-incentivized without crossing the threshold above which faithful transmission resumes; bandwidth can be restored only by altering the observational apparatus itself so that its capacity crosses the rate required by τ. The implications for current substrate-independent audit institutions are taken up in section 9.
5 The Rice Constraint — Semantic Reach and Bounded Auditability
The semantic reach dimension is the class of task-relevant properties of S that A’s description language and decision procedure can distinguish. Rice (1953) names a neighboring constraint: even where semantic properties of program behavior can be specified, non-trivial properties of that behavior generally outrun complete decision procedures. The audit consequence the present note draws is that any practical audit operates on a bounded subclass of task-relevant properties: the subclass within which L_A’s vocabulary and L_A’s decision procedure are jointly sufficient to render distinct judgments.
Rice’s theorem is adjacent to the present claim. It establishes, for sufficiently general program classes, that every non-trivial property of the function computed by a program is undecidable in general. The audit-side consequence concerns the absence of a complete decision procedure over the full task domain. Classification continues within that absence: when an auditee produces behavior whose task-relevant property lies outside the decision procedure available to A, A’s judgment cannot separate that behavior from a behavior whose task-relevant property is its complement. Two distinct conditions of S map to the same audit category in A.
This is the formal source of bounded auditability. The term names the shape of audit’s success. Practical audit instruments succeed by operating within a subclass of properties for which their language is expressive enough and their decision procedure is complete enough to render reliable distinctions; the instruments fail at the boundary of that subclass. Bounded auditability is therefore a positive condition: the description of where audit informativeness is sustained. It names the form audit takes when the audit task does not require distinctions that the auditor’s representational machinery cannot draw.
The boundary of bounded auditability is task-dependent. As R_{τ,e}(S) — the semantic reach the audit task requires — moves outward into properties of S whose representations A’s L_A cannot accommodate, the bounded subclass within which audit was informative shrinks, and judgments produced over the wider task domain begin to merge categories that the audit task was meant to keep separate. The 2008 forensic instance taken up in section 6 illustrates this drift: the AAA category continued to render distinct judgments for plain-vanilla corporate bonds while losing the capacity to distinguish AAA-rated structured-finance instruments whose underlying semantic properties had migrated outside what the rating language could express.
The constraint named here applies where audit proceeds from the auditee’s behavior; it is silent on the audit task that proceeds from inspection of the auditee’s internal mechanisms. Mechanistic inspection of model weights, for example, lies outside what Rice’s theorem governs. The structural route changes the evidentiary surface of the audit rather than the asymmetry it operates within. It asks the auditor to record weight states and activation patterns, to distinguish mechanism-level properties, and to keep the interpretability tool from collapsing into the same representational substrate as the system it interprets. The capacity vector is reweighted along these axes rather than escaped. Whether the mechanistic route improves on the behavioral route is a question about which dimension of capacity is most easily approximated for a given task; either route runs through the asymmetry the note maps.
A practical implication. Semantic reach deficits, unlike bandwidth deficits, can sometimes be approximated by procedural repair — through the introduction of new categories, new evaluation rubrics, new domain-specific languages developed as auditees evolve. The repair is partial: a new category installed within an existing audit language extends the bounded subclass without removing the underlying limit. The conditions under which procedural extension of L_A approximates a missing semantic reach, and the conditions under which it does not, are taken up further in section 9.
6 The 2008 Gaussian Copula Episode — A Staged Capacity Inversion
The note uses audit broadly: not only statutory inspection, but any institutional cycle in which one system certifies, rates, or renders another system legible for third-party reliance. Under this usage, credit rating agencies in the period from 2000 through 2008 conducted an audit of structured-finance instruments — a cycle of selecting observations from issuers, applying a description language calibrated to default correlation, and producing judgments whose institutional weight permitted third parties to act on those judgments without independent inquiry. The Gaussian-copula collapse documented at the level of system dynamics by Ortulanu (2026 §3) is here read as a forensic instance of capacity asymmetry, in five stages.
The staging is analytic rather than archival: the dates mark the emergence of capacity conditions as they become visible in the available record, not the first causal occurrence of each condition.
Stage one: coupled architectural inversion, in place by 2000. Moody’s Binomial Expansion Technique, introduced in December 1996 and dominant through 2003–2004 (Fender & Kiff, 2004), mapped the auditee’s collateral pool to a hypothetical portfolio of N uncorrelated equivalent assets via the Diversity Score. The mapping was lossy compression as a representational primitive: correlation, the parameter that investment-bank derivatives desks were modelling explicitly via the Gaussian copula by 2000 (Li, 2000), was collapsed into a single integer in the rating language. The method bought throughput by narrowing the correlation structure the rating language could see. The immediate loss was semantic reach: distinctions among correlation regimes the auditee’s language could draw were unavailable in L_A. Stage one is therefore a coupled inversion of the kind §2.4 describes in static form. The audit interface could record the auditee’s stream only by reducing its task-relevant dimensionality, and R_{τ,e}(S) exceeded the distinctions available in L_A before the operational pressures of stages three and four arrived. MacKenzie & Spears (2014) document the same gap in dynamic terms: investment banks ran multi-period correlated default models on computational grids, while rating agencies operated one-period methods that did not catch up until the second half of the decade.
Stage one was a latent capacity deficit rather than an immediate collapse. The rating interface continued to generate stable judgments so long as the correlation process remained sufficiently external to the instruments being rated. The deficit became active only when the auditee changed shape.
Stage two: self-reference dynamics activate, 2003–2006. The CDO bid for the BBB tranche of subprime residential mortgage-backed securities began reshaping mortgage origination. By 2005, CDO managers were absorbing a large share of new BBB tranches, and origination volumes at Countrywide, New Century, and similar lenders expanded as the CDO pipeline absorbed more of what they produced. The exogeneity assumption on which the rating language depended — that the correlation structure was external to the mortgage market — ceased to hold. The architectural deficit of stage one had not produced a visible failure on its own. The activation of self-reference dynamics gave it an auditee whose output stream now required exactly the expressive resources the rating language did not possess.
Stage three: operational bandwidth overload, 2004–2007. Moody’s CDO-related annual revenue rose from approximately $61 million in 2000 to over $260 million in 2006 (FCIC, 2011, ch. 8); CDO deal volume increased sevenfold over the same window while staffing did not expand at the same rate. Internal correspondence subpoenaed by the U.S. Senate Permanent Subcommittee on Investigations (2011, p. 6) and summarised by the SEC (2008) documented analysts working under conditions described in 2007 as “rushed, overworked and demoralized,” operating with what the SEC identified as out-of-date models and unclear ratings criteria. The first explicit correlation-aware rating-agency CDO methodologies arrive late in this window — Witt’s correlated binomial framework in August 2004, Moody’s revised structured-finance correlation assumptions in June 2005, S&P’s multi-period Evaluator v3.0 in December 2005. Each upgrade arrived after the auditee had moved further.
Stage four: semantic reach inversion, 2005–2007. The AAA category, stable for corporate bonds for decades, lost its capacity to render distinct judgments across the asset classes the rating apparatus had absorbed. ABS, CDO, ABS-CDO, and CDO² instruments shared a single category designation while sharing increasingly little of the discriminative content the category was meant to carry. By the end of the period, more than ninety percent of AAA-rated subprime residential mortgage-backed securities issued in 2006 and 2007 would be downgraded (Senate Permanent Subcommittee, 2011). The structural sensitivity of these instruments’ default probabilities to small errors in correlation assumptions, documented quantitatively by Coval, Jurek, and Stafford (2009), had not been a primary input to the rating language. Two structurally distinct conditions of the auditee received the same audit judgment: the bounded subclass within which the rating language remained informative had contracted to corporate bonds and similar instruments, while structured-finance instruments had migrated outside it.
Stage five: the audit cycle ceases to be informative, 2007–2008. The wave of downgrades that began in July 2007 marks the moment when the rating cycle’s outputs stopped tracking the task-relevant state of the auditee. Earlier judgments did not disappear; institutional repair did not, on its own, restore the cycle. The conditions the first three stages had assembled, and that stage four carried into the categorical layer, finally coincided.
The reading offered here departs from the standard one. MacKenzie (2011, pp. 1783–84, 1796) documents the organizational separation of ABS and CDO rating groups, and the resulting evaluation cultures, as the sociological substrate of the same collapse; the present note does not replace that account. The capacity frame applies one layer below: it concerns the information-theoretic conditions an audit must meet for its judgments to remain informative regardless of the cultures that staff it. Capacity asymmetry can persist in a single integrated organization that has overcome the cultural separation MacKenzie describes. The complementary frame of ratings shopping (Skreta & Veldkamp, 2009) treats complexity as a scalar that competition over multiple agencies makes gameable; the present account treats capacity as a vector along which a single auditor, with no competition at all, can fall below the requirements of its task. The capacity account commits to a sharp counterfactual. Had issuer-pays arrangements been replaced by a payment structure with aligned incentives, the architectural compression in the rating language would still have been present, and the self-reference dynamics of 2003–2006 would still have made it visible. Incentives explain why the simplification persisted. Capacity explains why the rating language as then constituted could not absorb what it was being asked to record.
7 LLM-as-Judge — Capacity Inversion in the Current AI Evaluation Substrate
The structural correspondence between the 2008 Gaussian-copula episode and the current evaluation substrate of frontier machine-learning systems concerns capacity geometry rather than identical mechanism. The auditor in the current case is, in a growing share of evaluation pipelines, another machine-learning system: the LLM-as-judge architecture documented by Zheng et al. (2023) and the model-based red-teaming, RLAIF, and constitutional procedures of which it is one form. The auditee is the model under evaluation. The audit task τ is the production of a ranking, score, or pass-fail judgment over the auditee’s outputs. Capacity asymmetry here is self-inflicted: auditor and auditee are drawn from a single representational lineage, and the auditor’s capacity vector is bounded by the same substrate that bounds the auditee’s.
The bias profile reported in the empirical literature on LLM-as-judge does not exhaustively reduce to the three dimensions named in section 2, but it clusters around them in a diagnostically useful way.
Bandwidth. The motivation for delegating evaluation to a model is, in the first instance, that human evaluation does not scale to the throughput at which the auditee produces outputs. The substitution amounts to selecting an auditor whose record-rate matches the auditee’s output-rate; under capacity asymmetry, this match is purchased by accepting compression of the audit signal into whichever features the model auditor is equipped to record.
Semantic reach. The biases catalogued by Zheng et al. (2023) and refined in subsequent work — preference for verbose responses (Saito et al., 2023) and sensitivity to position in pairwise comparisons (Wang et al., 2023) — are, on the present reading, indications of where the model auditor’s bounded subclass ends. When the auditee produces behavior whose task-relevant property is finer than the model auditor’s decision procedure can distinguish, the judgment falls back to features that lie inside the bounded subclass: surface fluency, length, position. The judgment shifts from the property τ asks about to the surface form through which that property is presented. The substitution is the form semantic-reach inversion takes when the audit task asks for distinctions the audit language cannot draw — a property of the asymmetry, surfacing here through the LLM-as-judge architecture.
Self-reference depth. When τ requires the auditor to stand outside the auditee’s representational lineage — as the substrate-independence requirement of the audit task does — a same-family or lineage-trained judge does not meet R_{τ,d}(S) by architecture alone. The judge can still produce rankings. What it cannot supply is the depth condition the criterion-forming step depends on. The self-enhancement bias documented for several LLM-as-judge configurations — the tendency of a judge to prefer outputs that resemble its own (Panickssery, Bowman, & Feng, 2024) — is the visible signature of this deficit: the audit criterion has partly collapsed into the object being audited.
The bias profile appears as a list of independent engineering problems but operates as the surface form of a single capacity inversion across three dimensions. The contemporary case admits structural mapping but not yet the five-stage forensic offered for the 2008 episode in section 6: the cycle’s loss of informativeness as a recognized event has not arrived in the same documented form. The contemporary marker is therefore displacement of the audit interface: human judgment increasingly audits the model judge’s summaries rather than the auditee’s behavior directly, and the bandwidth problem moves one level up.
The 2008 case differed in one further respect. The financial cycle’s collapse was visible because external reality — actual defaults on actual mortgages — eventually arrived to settle the discrepancy between the rating language and the auditee. The audit system received an external bill. The current evaluation substrate has no equivalent bill in the same form: where auditor and auditee are both model systems judging model-produced representations, the asymmetry can persist without producing a moment of recognized failure. Capacity asymmetry on the contemporary substrate may therefore present as detachment — the audit cycle continues to produce judgments, the judgments continue to be acted on, and the question whether they track anything outside the cycle becomes increasingly unanswerable from within it. Three markers would be diagnostic from inside the cycle: benchmark saturation as independent verification becomes harder; high inter-judge agreement on a contracting τ; and growing divergence between evaluation results and downstream external consequence. The arrival of any single recognized failure event — an external bill in the 2008 sense — would be the marker that detachment had ceased to be the form. Distinguishing those markers from background variation would require measurement protocols developed outside the present note.
What a contemporary stage-five would look like — whether it would arrive as a single recognized event of the kind July 2007 supplied for structured finance, or whether it would arrive instead as a diffuse loss of evaluation discriminability across benchmarks — is the empirical question the third note in this series takes up. Whether and how this inversion propagates through training-evaluation feedback loops — an adjacent dynamic Perdomo et al. (2020) name as performative prediction, in which predictions reshape the distributions they are meant to predict — is the further subject of that note. The present note locates the failure one layer out: in the audit relation’s capacity to remain informative as that reshaping proceeds.
8 The Externality Conditions, Reframed — A Duality with the First Note
The first note in this series identified three failure conditions under which evaluation categories lose informativeness: self-reference, anchor drift, and proxy displacement. It proposed three externality conditions as their structural negation: provenance independence, external anchoring, and speed parity. The two triads are presented there as system-level descriptions: the failure conditions describe what happens to the system being audited, and the externality conditions describe what an audit instrument must look like to block that failure.
The present note offers a second reading of the same triads. The two stand to one another as duals.
The first triad names the auditee-side phenomena that emerge when an audit cycle is informationally outpaced. The second names the auditor-side capability deficits that produce the same outcome. Proxy displacement and bandwidth are direct duals: both name the rate at which the audit cycle transmits information about its auditee, the first as a system phenomenon (the original audit question migrates into a category judgment because the running evaluation cannot keep pace), the second as an auditor capability (the rate at which the description language can record the task-relevant stream). Anchor drift and semantic reach are duals across the system–auditor axis: a category whose external reference has drifted is one that the auditor’s description language has lost the capacity to distinguish. Self-reference and self-reference depth are the third pair: a system whose outputs become inputs to its own categorization is one whose representational substrate the auditor cannot describe from outside.
The dual reading does not replace or supersede the system-level reading of the first note. It renders explicit a structure already implicit there: each capacity dimension named in the present note is what an auditor would have to possess in order for the corresponding system-level failure to fail to obtain. The two notes therefore describe one relation from two equally informative sides — the first from the auditee, the second from the auditor — and the first note’s externality conditions are recoverable from the present note’s framework as the system-level negations of capacity inversion along each dimension. Provenance independence is the institutional shape that, when achieved, addresses the self-reference depth requirement R_{τ,d}(S); external anchoring addresses R_{τ,e}(S); speed parity addresses R_{τ,b}(S). The structural derivation of substrate independence as the necessary form of the audit instrument when self-reference depth requirements cannot be met by governance alone is the subject of the fourth note in this series.
A note on the reflexive position of the argument. The framework also has a reflexive implication for its own review. The claim is structural, not personal: any review of this argument is itself an audit relation, and the conditions the argument names apply to it. The framework remains open to criticism by argument; what it specifies is the conditions under which such criticism is itself informative. Where the relevant capacity dimensions of the audit relation between argument and reviewer are met, the review proceeds informatively. Where they are not, the review’s judgments stand in the same structural position as any other audit judgment under capacity asymmetry.
9 Implications for Substrate-Independent Audit at Scale
The three externality conditions identified in the first note are not equally substitutable. Provenance independence can sometimes be approximated by governance structure: an organization separated from the auditee by formal independence rules approximates the self-reference depth requirement R_{τ,d}(S) without obtaining substrate-level independence. External anchoring can sometimes be approximated by language design: a benchmark taxonomy carefully developed to distinguish task-relevant properties approximates the semantic reach requirement R_{τ,e}(S) without resolving the underlying decision-procedure limit. Speed parity does not admit the same approximation. Bandwidth recovery requires a parallel observational apparatus capable of operating at the tempo and scale of the auditee itself; substituting governance or language design leaves R_{τ,b}(S) unchanged, and leaves the rate at which the audit cycle can record the task-relevant stream unchanged.
The reason lies in what each repair changes. Provenance independence is partly approximable through governance: organizational separation rules fix who forms the judgment, and the judgment-formation procedure is what the self-reference depth requirement constrains. External anchoring is partly approximable through language design: new categories extend L_A’s distinguishable subclass, fixing what the judgment can sort, and the available distinctions are what the semantic reach requirement constrains. Bandwidth has no parallel route. The rate constraint is exogenous to L_A’s vocabulary and to the criterion’s procedural form: the rate at which task-relevant signal can be recorded is set by the observational apparatus, not by the rules under which the apparatus operates. Procedural repair can change who judges and what categories the judgment uses. It cannot change how fast the judgment can be formed without enlarging the apparatus that forms it.
A reasonable objection: bandwidth deficit can be reduced by sampling, statistical inference, or hierarchical audit, even where direct rate parity is unavailable. The objection holds in part. Sampling can reduce R_{τ,b}(S) where the task-relevant failure modes are sparse, stable, and externally specifiable in advance. It does not reduce R_{τ,b}(S) where the failure modes evolve with the auditee’s own optimization process: the sampler must already know the class of behaviors whose emergence it is meant to detect, and that class is precisely what the audit task asks the audit to learn. The objection is sometimes posed in a stronger form: that audit, like decision problems in computational complexity, may admit verification structurally cheaper than generation. The asymmetry holds for tasks whose specification is given in advance. Frontier audit is not such a task — the specification of which behaviors must be detected is precisely what the audit cycle is asked to develop. The conditions under which sampling and ensemble methods can substitute for bandwidth parity, and the conditions under which they cannot, are taken up in the fourth note in this series.
The asymmetry has an empirical instance in the present landscape of frontier evaluation bodies (Anderljung et al., 2023). The UK AI Security Institute (AISI, renamed from AI Safety Institute in 2025) and the US Center for AI Standards and Innovation (CAISI, which replaced the US AI Safety Institute in June 2025) have approximated provenance independence by institutional separation from the laboratories they audit, and have formalized evaluation practice through frameworks such as the UK AISI’s Inspect framework (UK AI Safety Institute, 2024) and the NIST AI Risk Management Framework Generative AI Profile (NIST, 2024) — frameworks that approach the semantic reach the audit task requires. Both bodies operate through voluntary cooperation with the laboratories they evaluate, without legal mandate to access frontier-model outputs at the tempo of frontier-model development. The 2025 US reorganization shifted CAISI’s stated posture toward serving as “industry’s primary point of contact” under a lighter regulatory frame (U.S. Department of Commerce, 2025). The underlying voluntary structure did not change. The dimension neither body has been able to approximate is bandwidth: tracking frontier-laboratory outputs at that tempo would require an observational apparatus whose operating speed is much closer to a frontier laboratory than to a conventional regulator. The trade-off documented in the first note (Ortulanu, 2026 §6) — that the AISI institutional model achieves substrate externality “by paying the cost of speed” — is, on the analysis offered here, not merely contingent. The trade-off is the form the asymmetry takes when one of the three dimensions cannot be substituted by procedural repair.
The argument is narrow: it concerns the residue institutional design leaves untreated. Where the residue is small — where the audit task does not require bandwidth above what the auditor can sustain — institutional repair restores informativeness. Where the residue is structural — where bandwidth requirements grow with the auditee while the auditor’s bandwidth does not — institutional repair sustains the audit’s institutional form without sustaining its informativeness. Substrate-independent audit at the current frontier sits in the second of these conditions — the structural residue case.
The wider question of why the dominant framing of frontier-safety work positions capacity asymmetry outside the space of recognized problems was raised at the meta-level in the first note (Ortulanu, 2026 §7), and is taken up further in the seventh note in this series.
A final observation. The asymmetry the note names is a coupled set of constraints, and a deficit in one dimension propagates to deficits in the others. The propagation unfolded slowly across the years that documented the 2008 collapse. On the contemporary substrate, where the auditor and the auditee evolve together within recursive evaluation cycles, an architectural inversion in one dimension propagates to the others faster — the interval compresses. The 2008 episode unfolded over a decade. The contemporary episode, if it is one, would unfold on the cycle of model training rather than the cycle of credit-rating-agency methodology revision.
Until audit instruments meet the bandwidth, reach, and independence their tasks require, evaluations will continue to measure the limits of the audit apparatus rather than the state of the model.
The note does not propose a complete external instrument. It specifies the dimensions along which an instrument’s capacity must meet the auditee’s task-relative requirement, and identifies the dimension along which present institutional repair is structurally insufficient. What such an instrument would be made of, and at what scales it could be constructed, lie beyond what the present analysis can settle.
References
Anderljung, M., Barnhart, J., Korinek, A., Leung, J., O’Keefe, C., Whittlestone, J., et al. (2023). Frontier AI regulation: Managing emerging risks to public safety. arXiv:2307.03718.
Coval, J. D., Jurek, J. W., & Stafford, E. (2009). The economics of structured finance. Journal of Economic Perspectives, 23(1), 3–25.
Fender, I., & Kiff, J. (2004). CDO rating methodology: Some thoughts on model risk and its implications. BIS Working Paper No. 163. Bank for International Settlements.
Financial Crisis Inquiry Commission. (2011). The Financial Crisis Inquiry Report. U.S. Government Printing Office.
Gödel, K. (1931). Über formal unentscheidbare Sätze der Principia Mathematica und verwandter Systeme I. Monatshefte für Mathematik und Physik, 38, 173–198.
Li, D. X. (2000). On default correlation: A copula function approach. Journal of Fixed Income, 9(4), 43–54.
MacKenzie, D. (2011). The credit crisis as a problem in the sociology of knowledge. American Journal of Sociology, 116(6), 1778–1841.
MacKenzie, D., & Spears, T. (2014). ‘The formula that killed Wall Street’: The Gaussian copula and modelling practices in investment banking. Social Studies of Science, 44(3), 393–417.
NIST. (2024). AI Risk Management Framework: Generative AI Profile (NIST AI 600-1). National Institute of Standards and Technology.
Ortulanu, G. (2026). A category-level failure mode not captured by distribution metrics. osservadore.
Panickssery, A., Bowman, S. R., & Feng, S. (2024). LLM evaluators recognize and favor their own generations. arXiv:2404.13076.
Perdomo, J. C., Zrnic, T., Mendler-Dünner, C., & Hardt, M. (2020). Performative prediction. Proceedings of the 37th International Conference on Machine Learning, 119, 7599–7609.
Rice, H. G. (1953). Classes of recursively enumerable sets and their decision problems. Transactions of the American Mathematical Society, 74(2), 358–366.
Saito, K., Wachi, A., Wataoka, K., & Akimoto, Y. (2023). Verbosity bias in preference labeling by large language models. arXiv:2310.10076.
Shannon, C. E. (1948). A mathematical theory of communication. Bell System Technical Journal, 27(3), 379–423; 27(4), 623–656.
Skreta, V., & Veldkamp, L. (2009). Ratings shopping and asset complexity: A theory of ratings inflation. Journal of Monetary Economics, 56(5), 678–695.
UK AI Safety Institute. (2024). Inspect: An open-source framework for large language model evaluations.
U.S. Department of Commerce. (2025). Statement from U.S. Secretary of Commerce Howard Lutnick on transforming the U.S. AI Safety Institute into the Center for AI Standards and Innovation. Press release.
U.S. Securities and Exchange Commission. (2008). Summary report of issues identified in the Commission staff’s examinations of select credit rating agencies. SEC.
U.S. Senate Permanent Subcommittee on Investigations. (2011). Wall Street and the financial crisis: Anatomy of a financial collapse [Levin–Coburn Report]. United States Senate.
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., & Sui, Z. (2023). Large language models are not fair evaluators. arXiv:2305.17926.
Witt, G. (2004). Moody’s correlated binomial default distribution. Moody’s Investors Service.
Zheng, L., Chiang, W.-L., Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., Zhang, H., Gonzalez, J. E., & Stoica, I. (2023). Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. Advances in Neural Information Processing Systems, 36. arXiv:2306.05685.