27. High Availability

27.1 What Availability Means Here #

Availability in a correlation architecture is not a single property. The framework distinguishes four:

Table 56. Four availability types. They fail independently and warrant independent objectives. A common design error is specifying a single platform SLA, which either over-engineers serving or under-engineers correlation.

27.2 Multi-Region Topology #

flowchart TB
    subgraph GLOBAL["Global Control Plane"]
        direction LR
        GC1["Scope registry"]
        GC2["Policy bundles"]
        GC3["Schema registry"]
        GC4["Model registry"]
        GC5["Metadata catalog"]
    end

    subgraph RA["Region A — Active"]
        direction TB
        RA1["Ingestion<br/><i>active</i>"]
        RA2["Circulation log<br/><i>RF=3, rack-aware</i>"]
        RA3["Correlation<br/><i>active, checkpointed</i>"]
        RA4[("Graph hot<br/><i>causal cluster</i>")]
        RA5[("Lakehouse<br/><i>regional</i>")]
        RA6["Serving<br/><i>active</i>"]
        RA7["Autonomy Controller<br/><i>active</i>"]
    end

    subgraph RB["Region B — Active"]
        direction TB
        RB1["Ingestion<br/><i>active</i>"]
        RB2["Circulation log<br/><i>RF=3, rack-aware</i>"]
        RB3["Correlation<br/><i>active for B-partitioned<br/>entities</i>"]
        RB4[("Graph hot<br/><i>causal cluster</i>")]
        RB5[("Lakehouse<br/><i>regional</i>")]
        RB6["Serving<br/><i>active</i>"]
        RB7["Autonomy Controller<br/><i>standby</i>"]
    end

    subgraph DR["Region C — Recovery"]
        direction TB
        DR1["Cold infrastructure<br/><i>IaC, not running</i>"]
        DR2[("Log archive<br/><i>replicated</i>")]
        DR3[("Lakehouse replica<br/><i>cross-region</i>")]
    end

    GLOBAL -.-> RA & RB & DR
    RA2 <==>|"bidirectional mirror<br/>with origin tracking"| RB2
    RA5 ==>|"async replicate"| DR3
    RA2 ==>|"archive"| DR2

    NOTE["<b>Entity-partitioned active-active</b><br/>Each region owns a disjoint set of entity key ranges.<br/>Cross-region correlation is explicit, governed, and<br/>accepts higher latency."]

    RA -.- NOTE
    RB -.- NOTE

    classDef glob fill:#3a2540,stroke:#a878c0,stroke-width:2px,color:#f6eaff
    classDef act fill:#0a5570,stroke:#3fd0f0,stroke-width:2px,color:#ffffff
    classDef dr fill:#2a3340,stroke:#6b7a8c,color:#dbe4ee
    classDef note fill:#3d3313,stroke:#d4a636,color:#fff8e6
    class GLOBAL,GC1,GC2,GC3,GC4,GC5 glob
    class RA,RA1,RA2,RA3,RA4,RA5,RA6,RA7,RB,RB1,RB2,RB3,RB4,RB5,RB6,RB7 act
    class DR,DR1,DR2,DR3 dr
    class NOTE note

Figure 39. Multi-region high-availability topology. Regions are active-active but entity-partitioned: each region owns a disjoint set of entity key ranges and performs correlation only for its own. This avoids the fundamental problem of concurrent correlation over the same entities in two regions, which would produce duplicate and potentially conflicting correlations with no principled resolution.

27.2.1 Why Entity-Partitioned Active-Active #

The naïve active-active design — both regions correlating all entities — fails. Two regions independently correlating the same entity’s events will produce two correlations for the same relationship, with different confidence scores (because each saw a different subset under replication lag), and no principled way to reconcile them.

Entity partitioning avoids this: for any given entity, exactly one region is authoritative. Regions are both active, but over disjoint key ranges. On regional failure, the surviving region assumes the failed region’s key ranges by replaying from the mirrored log — which is why bidirectional mirroring with origin tracking is required rather than one-way replication.

27.3 Availability Tiers #

Table 57. Availability tiers with recovery objectives. The control plane is deliberately rated at the same level as the circulation log: correlation cannot proceed without scope definitions and policy decisions, so control plane unavailability halts the system’s core function even when every data-plane component is healthy.

27.3.1 Control Plane Availability #

The control plane is the framework’s most dangerous single point of dependency. The mitigations:

Local caching with TTL. Each correlation worker caches scope definitions and compiled policy bundles locally. On control plane unavailability, workers continue on cached configuration for a bounded period.

Fail-safe defaults on expiry. When the cache TTL expires and the control plane remains unavailable, workers do not continue on stale policy. They halt correlation for scopes whose policy cannot be confirmed. This is a deliberate choice of correctness over availability: continuing to form correlations under possibly-revoked policy is a compliance failure, and Principle 8 states that policy prevails absolutely.

Read-only degradation. The control plane may be available for reads while unavailable for writes. This is a fully acceptable degraded state — existing scopes continue, new scopes cannot be declared.

27.4 Failure Scenarios #

Table 58. Failure scenarios and their correlation integrity impact. The recurring theme is that replay closes gaps provided log retention exceeds the outage duration — which makes retention configuration an availability decision, not merely a cost decision.

27.5 Consistency Posture #

Per Chapter 4, the framework’s stated position:

Correlations are eventually consistent by default. A correlation formed in one region becomes visible in another after replication. Consumers must tolerate this, and the framework requires that correlation reads expose the freshness of the data they reflect.

Strong consistency by exception. Correlations gating irreversible autonomous action (L4 and above) MUST be read with strong consistency, accepting the latency and partition-tolerance cost. In practice this means such reads are served from the authoritative region for that entity, and fail rather than serve stale data under partition.

Read-your-writes for human workflows. An analyst who confirms a correlation must see the confirmation reflected immediately. Session-scoped consistency is provided for interactive workflows.

27.6 Operational Practices #

Table 59. Operational availability practices. The degradation simulation is specific to this architecture and is the drill most likely to reveal a serious defect, because the absence–degradation confound (Chapter 19) fails catastrophically and silently until it is tested.

Next: 28. Performance Considerations →