26. Scalability

26.1 The Dimension That Matters #

Conventional data platforms scale along a familiar axis: data volume. A Data Swimming Pool has a second axis that dominates it.

An organization can double its event volume and add roughly double the storage and ingestion capacity. If it doubles the number of domains participating in correlation, the number of potential cross-domain relationships grows quadratically, and if scopes are declared naïvely the cost grows with it.

This is why Principle 3 exists, and why scalability in this framework is primarily a governance problem expressed as an engineering one.

26.2 Scaling Dimensions #

Table 53. Scaling dimensions and strategies. Two dimensions exhibit non-linear growth and require architectural rather than capacity responses: potential correlation grows quadratically in event types, and traversal cost grows exponentially in depth. Both are addressed by hard limits rather than by adding resources.

26.3 Partitioning Strategy #

Partitioning determines whether correlation is cheap or expensive, and the framework’s strategy is layered.

flowchart TB
    subgraph L1P["Level 1 — Entity Partitioning"]
        direction TB
        E1["Partition by resolved entity key"]
        E2["<b>Effect:</b> all events for an entity<br/>are co-located → M2 correlation<br/>is local, no shuffle"]
        E3["<b>Risk:</b> hot keys — a single<br/>high-activity entity overloads<br/>one partition"]
        E4["<b>Mitigation:</b> composite key with<br/>salt for known hot entities;<br/>hot-key detection and split"]
    end

    subgraph L2P["Level 2 — Temporal Partitioning"]
        direction TB
        T1["Partition state by time bucket"]
        T2["<b>Effect:</b> window expiry is a<br/>bucket drop, not a scan"]
        T3["<b>Benefit:</b> O(1) expiry instead<br/>of O(n) eviction"]
    end

    subgraph L3P["Level 3 — Scope Isolation"]
        direction TB
        S1["Each scope runs as an<br/>independent job"]
        S2["<b>Effect:</b> scope failure is<br/>contained; scope cost is<br/>directly attributable"]
        S3["<b>Cost:</b> duplicated event<br/>consumption across scopes"]
        S4["<b>Mitigation:</b> co-schedule scopes<br/>sharing event types"]
    end

    subgraph L4P["Level 4 — Tenant / Domain Lanes"]
        direction TB
        D1["Physical isolation where<br/>regulation or residency requires"]
        D2["<b>Effect:</b> compliance boundary<br/>is also a failure boundary"]
        D3["<b>Cost:</b> cross-lane correlation<br/>requires explicit, governed bridge"]
    end

    L1P --> L2P --> L3P --> L4P

    classDef p fill:#0d4159,stroke:#22b3d6,stroke-width:1.5px,color:#eaf9ff
    class L1P,E1,E2,E3,E4,L2P,T1,T2,T3,L3P,S1,S2,S3,S4,L4P,D1,D2,D3 p

Figure 38. Four-level partitioning strategy. Entity partitioning makes the dominant correlation modality local. Temporal bucketing converts window expiry from an O(n) eviction scan into an O(1) bucket drop. Scope isolation makes cost attributable and failures contained. Domain lanes align the compliance boundary with the failure boundary.

26.3.1 The Hot Key Problem #

Entity partitioning has a characteristic failure: a small number of entities generate a disproportionate share of events. A corporate account with ten thousand subsidiaries, a shared device, a default null entity that resolution failures collapse into.

The framework’s mitigations, in order of preference:

1. Detect and split. Monitor per-key event rates; automatically apply a salt to keys exceeding a threshold, accepting that correlation for those keys becomes a two-stage aggregation.
2. Reject the null attractor. Resolution failures MUST NOT collapse into a shared “unknown” entity, which becomes an artificial hub correlating everything with everything. Unresolved events receive distinct placeholder identities and are excluded from entity-modality correlation until adjudicated.
3. Cap per-entity state. A single entity’s window state is bounded; exceeding it sheds oldest content and records that recall was reduced for that entity.

26.4 Elasticity #

Table 54. Elasticity by component. The correlation tier is the framework’s least elastic component: rescaling a stateful job requires redistributing large keyed state, which takes minutes rather than seconds. Capacity planning must therefore anticipate load rather than react to it.

26.5 Cost Control #

Cost in this framework is dominated by three items: correlation state, correlation compute, and graph storage. All three are governed by scope declarations, which is why cost control is a governance function.

Cost attribution. Every scope carries its cost, computed from its state footprint, its compute consumption, and its emission volume. Cost is reported per scope, per owner, per domain — which makes the conversation about whether a scope is worth its cost possible.

Cost per outcome. The framework recommends tracking cost per acted-upon insight rather than cost per correlation. A scope producing a million correlations of which none is acted upon has infinite cost per outcome and should be retired regardless of how cheap its correlations are individually.

Tiering. Recent correlations live in the graph hot tier; older ones move to the lakehouse. Recent events live on local disk; older ones move to object storage. Both transitions are automated and both are substantial cost levers.

Modality economics. Enabling all five modalities on every scope is the most common and most expensive design error. M3 (causal) in particular is orders of magnitude more expensive than M2 (entity) and should be enabled only where causal hypotheses are genuinely required and will genuinely be used.

26.6 Scale Limits and Honest Boundaries #

Table 55. Estimated practical scale boundaries. The most important row is active correlation scopes: the framework’s governance capacity — the ability to review, calibrate, and retire scopes meaningfully — fails well before its technical capacity does. An organization that cannot govern fifty scopes will not be rescued by infrastructure that can execute two hundred.

Next: 27. High Availability →