14. Correlation Engine

The Correlation Engine is the framework’s principal contribution. Everything preceding it prepares data for correlation; everything following it consumes correlations. This chapter specifies its internal architecture, its five detection modalities, its confidence semantics, its scope governance, and — with equal seriousness — its failure modes.

14.1 Purpose and Position #

Its position is deliberate. It sits above the circulation layer, so it observes all canonical events regardless of source. It sits below the intelligence tier, so rules, models, and generative systems consume its output rather than re-deriving relationships independently. This positioning is the architectural realization of Principle 1: correlation is infrastructure.

14.2 Internal Architecture #

flowchart TB
    IN[("canonical.* events<br/>from circulation layer")]

    subgraph GOV["Scope Governor"]
        direction TB
        SG1["Scope registry<br/><i>declared, versioned, reviewed</i>"]
        SG2["Budget enforcement<br/><i>state · compute · emission rate</i>"]
        SG3["Priority arbitration"]
        SG4{"Policy gate<br/><i>purpose · sensitivity</i>"}
    end

    subgraph MOD["Detection Modalities — parallel"]
        direction TB
        M1["<b>M1 Temporal</b><br/>windowed co-occurrence<br/>sequence patterns (CEP)<br/>lag-correlation"]
        M2["<b>M2 Entity</b><br/>shared resolved entity<br/>graph proximity<br/>shared attribute clusters"]
        M3["<b>M3 Causal</b><br/>Granger-style precedence<br/>intervention analysis<br/>structural hypotheses"]
        M4["<b>M4 Semantic</b><br/>embedding proximity<br/>ontology inference<br/>text/topic alignment"]
        M5["<b>M5 Behavioural</b><br/>baseline deviation co-occurrence<br/>learned pattern matching<br/>anomaly co-incidence"]
    end

    subgraph SCORE["Confidence Pipeline"]
        direction TB
        C1["Modality raw score"]
        C2["Evidence strength<br/><i>volume · consistency · specificity</i>"]
        C3["Prior adjustment<br/><i>base rate of relationship type</i>"]
        C4["Multiple-testing correction<br/><i>FDR control per scope</i>"]
        C5["Calibration mapping<br/><i>from historical outcomes</i>"]
        C6{"Threshold<br/>gate"}
        C1 --> C2 --> C3 --> C4 --> C5 --> C6
    end

    subgraph LIFE["Lifecycle Manager"]
        direction TB
        L1["Version + supersede"]
        L2["Temporal validity + expiry"]
        L3["Revision on late data"]
        L4["Auto-deprecation<br/><i>chronic under-performers</i>"]
    end

    OUT1[("Correlation store<br/>graph + lakehouse")]
    OUT2[("correlation.* topic")]
    FB["Outcome feedback<br/>from decision layer"]

    IN --> GOV
    SG4 -->|"permit"| MOD
    SG4 -->|"deny"| DENY["Audit + reject<br/><i>correlation not formed</i>"]
    MOD --> SCORE
    C6 -->|"above threshold"| LIFE
    C6 -->|"below threshold"| DISC["Discard + count<br/><i>recall telemetry</i>"]
    LIFE --> OUT1 & OUT2
    FB --> C5
    FB --> L4

    classDef gov fill:#3a2540,stroke:#a878c0,stroke-width:2px,color:#f6eaff
    classDef mod fill:#0a5570,stroke:#3fd0f0,stroke-width:2px,color:#ffffff
    classDef score fill:#14415c,stroke:#4fb8d8,stroke-width:1.5px,color:#eaf6fb
    classDef life fill:#183028,stroke:#4caf7d,color:#e6fff2
    classDef bad fill:#4a2020,stroke:#d07070,color:#ffeaea
    classDef store fill:#0d4159,stroke:#22b3d6,stroke-width:2px,color:#eaf9ff
    class GOV,SG1,SG2,SG3,SG4 gov
    class MOD,M1,M2,M3,M4,M5 mod
    class SCORE,C1,C2,C3,C4,C5,C6 score
    class LIFE,L1,L2,L3,L4 life
    class DENY,DISC bad
    class IN,OUT1,OUT2 store

Figure 19. Correlation Engine internal architecture. Four subsystems: the Scope Governor enforces bounded scope and policy before any detection occurs; five detection modalities operate in parallel; a six-stage confidence pipeline including explicit multiple-testing correction and outcome-based calibration; and a lifecycle manager handling versioning, expiry, revision, and automatic deprecation. Note that the policy gate precedes detection — a prohibited correlation is never formed, not merely never shown.

14.3 The Five Detection Modalities #

M1 — Temporal Correlation #

Mechanism. Detects relationships based on co-occurrence, sequence, or lag in event time.

Three sub-mechanisms:

• Windowed co-occurrence. Events of declared types occurring within a window for the same or related entities. The simplest and cheapest modality.
• Sequence pattern matching. Ordered patterns with optional negation and timing constraints, implemented via CEP automata (NFA-based, following SASE [14] and Flink CEP). Example: authentication from new device → within 10 minutes → payee added → within 5 minutes → high-value transfer, with no step-up authentication in between.
• Lag correlation. Statistical association between two time series at a discovered lag, useful for relationships where the delay is unknown a priori — for example, supplier batch change to defect rate.

Strengths. Cheap, interpretable, directly actionable. Sequence patterns are highly precise when the pattern is genuinely characteristic.

Weaknesses. Co-occurrence is the weakest form of evidence. In a high-volume environment, unrelated events co-occur constantly. Without base-rate correction, temporal co-occurrence generates overwhelming false positive volume.

M2 — Entity Correlation #

Mechanism. Relates events sharing a resolved entity, or entities within a bounded graph distance.

• Shared entity. The strongest and cheapest correlation: two events concern the same resolved commercial entity. Available essentially for free as a consequence of ingestion-time resolution and entity-key partitioning.
• Graph proximity. Events concerning entities within k hops in the knowledge graph — same household, same corporate group, same device, same beneficial owner.
• Shared attribute clusters. Events whose entities share a distinguishing attribute — same device fingerprint, same shipping address, same IBAN — which is the foundational mechanism for detecting coordinated behaviour such as fraud rings.

Strengths. High precision when resolution is sound. Computationally efficient. Highly explainable.

Weaknesses. Entirely dependent on entity resolution quality. An over-merged entity produces false correlations at scale — the hazard identified in Chapter 13. Graph proximity requires careful k bounding, since at k=4 or beyond in a dense graph, nearly everything is proximate to everything.

M3 — Causal Correlation #

Mechanism. Attempts to establish directional influence rather than mere association.

• Precedence testing. Granger-style tests establishing that one series improves prediction of another beyond the second’s own history.
• Natural experiment exploitation. Where an intervention occurred — a policy change, a deployment, a supplier switch — comparing pre- and post-periods, or treated and untreated cohorts.
• Structural hypothesis testing. Evaluating a declared causal graph against observational data, using do-calculus-informed methods.

Strengths. Highest decision value when correct. Directs remediation rather than merely flagging.

Weaknesses. Most computationally expensive, most assumption-dependent, most easily misinterpreted by consumers. Requires the largest evidence base.

M4 — Semantic Correlation #

Mechanism. Relates events by meaning rather than by key or timing.

• Embedding proximity. Events, documents, and entity descriptions embedded into a shared vector space; relationships inferred from nearest-neighbour proximity via HNSW or IVF indices [25].
• Ontology inference. Relationships derived by reasoning over a declared ontology — if A is a subtype of B and B relates to C, then A relates to C.
• Cross-lingual and cross-modal alignment. Relating a support call transcript to a technician’s written note to a product defect classification, despite no shared vocabulary.

Strengths. Discovers relationships invisible to key-based or timing-based methods. Essential for unstructured content. Effective across languages and modalities.

Weaknesses. Embedding proximity is not relationship — semantic similarity frequently reflects topical overlap rather than a genuine connection between the specific instances. Embeddings drift as models are updated, making historical comparisons unreliable unless embedding version is pinned. Explainability is weak: “these vectors are close” is a poor justification to present to an auditor.

M5 — Behavioural Correlation #

Mechanism. Relates events by their deviation from established baselines.

• Co-incident anomaly. Two entities or systems simultaneously departing from their respective baselines. Individually unremarkable deviations; jointly significant.
• Learned pattern matching. Sequence or graph models trained on historically-confirmed relationships, applied to detect similar structures.
• Cohort divergence. An entity behaving unlike its peer cohort, where the cohort itself is behaviourally defined rather than declared.

Strengths. Detects novel patterns not previously declared — the modality most capable of finding what nobody thought to look for. Robust to adversarial adaptation, since baselines shift with genuine behaviour.

Weaknesses. Requires substantial baseline history before becoming useful (typically weeks to months per entity). Highly sensitive to baseline drift and to seasonality. Poor explainability. Vulnerable to poisoning if an adversary can influence the baseline period.

14.3.1 Modality Comparison #

Table 22. Correlation modalities compared. The framework’s guidance is that M2 (entity) should carry the majority of production correlation volume, because it combines low cost, high precision, and strong explainability. M1 provides breadth, M4 reaches unstructured content, M5 finds novelty, and M3 should be used sparingly and always as hypothesis.

14.3.2 Multi-Modal Reinforcement #

The framework’s most important scoring property: correlations confirmed by multiple independent modalities are substantially more trustworthy than any single-modality correlation.

flowchart TB
    E1["Event A<br/><i>payment declined</i>"]
    E2["Event B<br/><i>support call</i>"]

    E1 & E2 --> M1C["<b>M1 Temporal</b><br/>within 8 minutes<br/>raw score 0.35"]
    E1 & E2 --> M2C["<b>M2 Entity</b><br/>same resolved customer<br/>raw score 0.90"]
    E1 & E2 --> M4C["<b>M4 Semantic</b><br/>call transcript mentions<br/>payment failure<br/>raw score 0.72"]
    E1 & E2 --> M5C["<b>M5 Behavioural</b><br/>no baseline deviation<br/>raw score 0.10"]

    M1C & M2C & M4C & M5C --> FUSE["<b>Evidence Fusion</b><br/>independent-modality reinforcement<br/>3 of 4 modalities concur"]
    FUSE --> BASE["Base-rate adjustment<br/><i>P(support call | payment decline)</i><br/>historically 0.31"]
    BASE --> FDR["FDR correction<br/><i>scope performed 4.2M tests</i>"]
    FDR --> CAL["Calibration mapping<br/><i>historical precision at this<br/>raw score: 0.88</i>"]
    CAL --> FINAL["<b>Final confidence 0.88</b><br/>type: SERVICE_RESPONSE_TO_EVENT<br/>evidence: 4 modality assessments<br/>validity: 24h from event B"]

    classDef ev fill:#1b2f3d,stroke:#5b8fb0,color:#e6f2fa
    classDef mod fill:#0a5570,stroke:#3fd0f0,color:#ffffff
    classDef proc fill:#14415c,stroke:#4fb8d8,color:#eaf6fb
    classDef fin fill:#183028,stroke:#4caf7d,stroke-width:2px,color:#e6fff2
    class E1,E2 ev
    class M1C,M2C,M4C,M5C mod
    class FUSE,BASE,FDR,CAL proc
    class FINAL fin

Figure 20. Confidence scoring pipeline, illustrated for a single correlation. Four modalities assess the same event pair independently. Evidence fusion rewards concurrence across independent modalities. Base-rate adjustment prevents the system from treating a common co-occurrence as remarkable. False discovery rate correction accounts for the millions of tests the scope performed. Final calibration maps the raw score to observed historical precision, so that a stated confidence of 0.88 means the correlation type has historically been correct 88% of the time at this score.

The calibration step in Figure 20 is what gives the confidence score meaning. Without it, a score of 0.88 is an arbitrary number produced by an internal formula. With it, the score is an empirical claim about historical precision that can be verified and challenged.

14.4 Scope Governance #

Principle 3 requires bounded scope. The Scope Governor enforces it.

14.4.1 Scope Declaration #

A correlation scope is a versioned, reviewed configuration artifact declaring:

Table 23. Correlation scope declaration elements. The Scope Governor refuses to execute a scope missing any mandatory element. Expected precision is the field that gates autonomy: a scope that cannot demonstrate its declared precision cannot be promoted to a higher autonomy level.

14.4.2 The Combinatorial Discipline #

14.4.3 Runtime Enforcement #

The Scope Governor monitors and acts:

• State budget exceeded: warn at 80%, shed oldest window content at 95%, suspend scope at 100% with alert.
• Emission rate cap exceeded: raise the effective confidence threshold dynamically until the rate conforms, and record that recall was sacrificed.
• Precision below declared expectation: automatically demote the scope’s autonomy level and notify the owner. This is the framework’s most important automated safeguard.
• Review overdue: mark the scope stale; stale scopes are excluded from autonomy levels 3 and above.

14.5 Correlation Lifecycle #

stateDiagram-v2
    [*] --> Candidate: modality detects
    Candidate --> Rejected: below threshold
    Candidate --> Denied: policy gate refuses
    Candidate --> Provisional: above threshold,<br/>window open
    Provisional --> Confirmed: window closed,<br/>batch reconciled
    Provisional --> Revised: late data arrives
    Revised --> Confirmed: re-evaluated
    Provisional --> Retracted: contradicted by<br/>late evidence
    Confirmed --> Validated: outcome feedback<br/>confirms prediction
    Confirmed --> Falsified: outcome feedback<br/>contradicts
    Confirmed --> Expired: temporal validity<br/>elapsed
    Confirmed --> Superseded: re-correlation (W2)<br/>produces new version
    Validated --> Expired: temporal validity<br/>elapsed
    Falsified --> Deprecated: pattern chronically<br/>under-performs
    Deprecated --> [*]: scope disabled
    Expired --> [*]: archived
    Retracted --> [*]: archived + consumers notified
    Rejected --> [*]: counted only
    Denied --> [*]: audited only

    note right of Provisional
        Actionable at autonomy L0–L2 only.
        Irreversible action prohibited
        until Confirmed.
    end note

    note right of Falsified
        Feeds calibration.
        Repeated falsification of a
        correlation TYPE triggers
        automatic deprecation (P10).
    end note

Figure 21. Correlation lifecycle state machine. Two constraints are architecturally enforced: provisional correlations may drive only reversible actions (autonomy levels 0–2), and correlation types that are chronically falsified by outcome feedback are automatically deprecated rather than persisting indefinitely as accumulated noise.

14.6 Failure Modes and Mitigations #

Table 24. Correlation Engine failure modes. Confidence miscalibration is the most insidious, because the system continues to operate and produce plausible output while its stated confidence has silently ceased to mean anything. It is detectable only by deliberate measurement against outcomes.

14.7 Relationship to Complex Event Processing #

The framework’s relationship to CEP [13] warrants precise statement, since CEP is its closest antecedent.

What the framework takes from CEP: the event pattern abstraction, NFA-based sequence matching, event hierarchies and abstraction, and the notion of derived higher-order events. The M1 sequence sub-mechanism is essentially a CEP engine, and the framework recommends using a mature one rather than reimplementing.

What the framework adds:

Table 25. Correlation Engine relative to classical CEP. The framework is fairly characterized as CEP generalized along four axes: from declared to discovered, from transient to persistent, from binary to calibrated, and from application-scoped to platform-governed.

Next: 15. Rules Engine →