8. Introducing Data Swimming Pool

8.1 Definition #

The definition is dense by intent. Each clause carries architectural weight:

8.2 The Metaphor Formalized #

Chapter 3 introduced the metaphor. Here it is mapped onto architecture explicitly, because the mapping is the design.

Table 9. The swimming pool metaphor mapped to architectural commitments. The metaphor is a design tool rather than a rhetorical device: each property corresponds to a specific, chapter-level design decision.

Three properties deserve emphasis because they distinguish this framework from a data lake most sharply.

Finite, engineered capacity. A lake’s implicit promise is unbounded retention. A pool’s explicit property is designed capacity. This maps to a hard architectural rule: correlation scope MUST be bounded. The framework does not attempt to correlate everything with everything. It correlates within declared, budgeted, and reviewable scopes. Chapter 14 treats the combinatorial consequences of violating this rule.

Continuous maintenance. A pool left unmaintained for a month becomes unusable. So does a correlation substrate: entity resolutions decay, confidence calibrations drift, relationship types become stale, and semantic embeddings age. The framework treats maintenance as an operational requirement with an owner and a budget, not as a project phase.

Marked depth. Swimmers are told where the water becomes dangerous. The Autonomy Ladder performs the same function for machine decisions, and it is enforced by the architecture rather than by policy documents.

8.3 Conceptual Model #

flowchart TB
    subgraph INFLOW["INFLOW — Heterogeneous Sources"]
        direction LR
        I1["Transactional<br/>OLTP · CDC"]
        I2["Streaming<br/>IoT · Clickstream · Logs"]
        I3["Batch<br/>Files · Extracts · Partner feeds"]
        I4["External<br/>Market · Weather · Registry"]
        I5["Unstructured<br/>Documents · Voice · Image"]
    end

    subgraph POOL["THE POOL — Active Medium"]
        direction TB
        FILT["<b>Filtration</b><br/>contracts · quality · dedup · entity resolution · classification"]
        CIRC["<b>Circulation</b><br/>canonical event model · streaming + batch substrates · replayable log"]
        INTER["<b>Interaction</b><br/><i>Correlation Engine</i><br/>temporal · entity · causal · semantic · behavioural"]
        CHEM["<b>Chemistry</b><br/>confidence calibration · drift detection · edge expiry"]
        DEPTH["<b>Depth Control</b><br/>Autonomy Ladder L0–L5 · human control points"]

        FILT --> CIRC --> INTER --> CHEM
        CHEM -.recalibrate.-> INTER
        INTER --> DEPTH
    end

    subgraph SUBSTRATE["PERSISTENT SUBSTRATE"]
        direction LR
        KGRAPH[("Knowledge Graph<br/>entities · events · relationships")]
        LAKEH[("Lakehouse<br/>raw · refined · curated")]
        VEC[("Vector Index<br/>semantic embeddings")]
        META[("Metadata Store<br/>lineage · provenance · policy")]
    end

    subgraph OUTFLOW["OUTFLOW — Governed Intelligence"]
        direction LR
        O1["Insights<br/><i>versioned objects</i>"]
        O2["Alerts<br/><i>correlated, suppressed</i>"]
        O3["Predictions<br/><i>context-enriched</i>"]
        O4["Recommendations<br/><i>evidence-backed</i>"]
        O5["Actions<br/><i>bounded, reversible</i>"]
    end

    LIFE["<b>OVERSIGHT</b><br/>Governance · Security · Privacy · Human-in-the-loop"]

    INFLOW ==> POOL
    POOL <==> SUBSTRATE
    POOL ==> OUTFLOW
    OUTFLOW -.outcome feedback.-> INTER
    LIFE -.enforces.- POOL
    LIFE -.enforces.- OUTFLOW

    classDef inflow fill:#1b3a4b,stroke:#4f9fc4,stroke-width:1.5px,color:#e8f4fa
    classDef pool fill:#0b3d5c,stroke:#22b3d6,stroke-width:2px,color:#eaf6fb
    classDef sub fill:#183028,stroke:#4caf7d,stroke-width:1.5px,color:#e6fff2
    classDef out fill:#3d3313,stroke:#d4a636,stroke-width:1.5px,color:#fff8e6
    classDef life fill:#3a2540,stroke:#a878c0,stroke-width:2px,color:#f6eaff
    class INFLOW,I1,I2,I3,I4,I5 inflow
    class POOL,FILT,CIRC,INTER,CHEM,DEPTH pool
    class SUBSTRATE,KGRAPH,LAKEH,VEC,META sub
    class OUTFLOW,O1,O2,O3,O4,O5 out
    class LIFE life

Figure 8. The Data Swimming Pool conceptual model. Heterogeneous inflow passes through filtration into a circulating medium where the Correlation Engine drives interaction between sources. Chemistry — confidence calibration and edge expiry — maintains the medium’s quality continuously. Depth control governs how much decision authority the system may exercise. Outflow is a spectrum of governed intelligence products whose outcomes feed back as supervision signal.

8.4 The Three Defining Objects #

The framework introduces three object types that have no direct equivalent in incumbent architectures. Their formal schemas appear in Appendix C.

8.4.1 The Canonical Event #

Every fact entering the pool — a transaction, a sensor reading, a document upload, a CDC row change, a batch record — is normalized into a canonical event carrying a stable identity, event-time and ingestion-time, a resolved entity reference set, a source lineage pointer, a policy classification, and a payload.

The critical field is the resolved entity reference set. A raw transaction references a numeric account code. The canonical event references the resolved commercial entity and retains the original identifier. This single normalization is what makes cross-domain correlation tractable, and performing it once at ingestion rather than repeatedly at query time is the primary mechanism by which the framework eliminates the correlation tax.

8.4.2 The Correlation #

A correlation is a first-class, persistent artifact asserting a typed relationship between two or more canonical events or entities. It carries:

• Type — one of the taxonomy in Chapter 19
• Modality — which of the five detection mechanisms produced it
• Confidence — a calibrated score with a stated interpretation
• Evidence set — the specific events and features supporting it
• Temporal validity — when it holds, and when it expires
• Provenance — the engine version, model version, and parameters that produced it
• Policy classification — purposes for which it may lawfully be used
• Supersession — a link to any prior correlation it revises

A correlation is not a query result. It is a governed asset with a lifecycle.

8.4.3 The Insight Object #

An Insight Object is a durable, versioned assertion about the business, derived from correlations, carrying its full evidence chain, an explicit confidence statement, a stated recommended action, an autonomy level, a permission set, and a lifecycle status (proposed, validated, acted upon, superseded, retracted).

This is a significant departure. In incumbent architectures the analytical output is ephemeral: a chart, an alert, a score. The framework’s position is that if a conclusion is worth acting on, it is worth persisting, versioning, attributing, and auditing — and that the inability to do so is why organizations cannot reconstruct why a decision was made six months later.

8.5 Data Lake versus Data Swimming Pool #

Table 10. Data Lake versus Data Swimming Pool. The final three rows are as important as the first thirteen: the framework carries a distinct failure mode, a substantially higher operational burden, and no deployment history.

8.6 What the Framework Does Not Claim #

To forestall predictable misreadings:

• It does not claim to replace the lakehouse. It assumes one as its persistence substrate.
• It does not claim to replace data mesh. It is proposed as a platform capability operable within a mesh.
• It does not claim that correlation is always valuable. Chapter 7 identifies the organizational profile for which it is material and states plainly that most organizations fall outside it.
• It does not claim causality. Correlations are hypotheses with confidence scores, bounded by the limits Lamport established in 1978.
• It does not claim autonomy is desirable by default. The Autonomy Ladder exists to constrain autonomy, and Levels 4 and 5 are recommended only under narrow conditions.
• It does not claim production readiness. This is its most significant limitation and is rated as such in every comparison in this document.

Next: 9. Core Design Principles →