Internal-External Representation Projection Diagnostics

Structural Problem

Interpretability methods attempt to map a model's internal computations onto human-understandable explanations. The structural problem is that this mapping is a projection — a dimensionality reduction from the high-dimensional internal representation space to the lower-dimensional explanation space — and projections inherently lose information. The question is not whether information is lost, but what information is lost, whether the loss is acceptable for the intended purpose, and whether the explanation preserves the structurally important properties of the internal computation.

Current interpretability approaches often assume that explanation fidelity can be improved incrementally. The structural perspective reveals that certain aspects of internal computation may be fundamentally non-projectable — they exist in dimensions of the internal space that have no counterpart in the explanation space, creating irreducible interpretability limits.

System Context

This application operates at the interface between model internals and human understanding, addressing interpretability methods, explanation systems, and audit requirements. The relevant system boundary includes the model's internal representation space, the explanation or interpretability method, the target explanation space, and the decisions that depend on explanation accuracy.

Diagnostic Capability

• Projection gap analysis quantifying the information lost when internal representations are projected onto explanation spaces
• Interpretability limit characterization identifying aspects of internal computation that are structurally non-projectable
• Explanation fidelity assessment evaluating whether specific interpretability methods preserve structurally important properties
• Audit sufficiency analysis determining whether available explanations provide adequate structural insight for governance requirements

Typical Failure Modes

• Explanation confabulation where the interpretability method produces plausible but structurally inaccurate explanations
• Selective projection where explanations accurately represent some aspects while systematically omitting others
• Fidelity illusion where explanation metrics suggest high accuracy while structurally important information is lost

Example Use Cases

• Interpretability method evaluation: Structural assessment of whether a proposed interpretability approach captures the relevant internal properties
• Audit framework design: Determining what level of internal transparency is structurally achievable and what governance decisions it can support
• Explanation system design: Structural guidance for building explanation systems that preserve the most decision-relevant internal properties

Strategic Relevance

As AI regulation increasingly requires model interpretability, understanding the structural limits of explanation becomes essential. Organizations need to know what can and cannot be explained, and to design governance frameworks that account for these structural limits rather than assuming that all internal behavior is interpretable.

SORT Structural Lens

The SORT framework addresses this application through four structural dimensions, each providing a distinct analytical layer.