Data Swimming PoolWhitepaper · Version 1.0
Source Download PDF 55 pages (333 pages) · 7 MB

Home / Front Matter

Index of Figures and Tables

Front Matter·1 min read

Index of Figures and Tables


Definition

This index is generated from the manuscript. Figures, tables, and listings are numbered sequentially across the whole document, independent of chapter number. Executive Summary items carry an ES- prefix; appendix listings carry a C- prefix.

Figures #

# Caption Chapter
Figure S-1 The idea in five steps Start Here — The Simple Version
Figure ES-1 Data Swimming Pool — high-level architecture 1. Executive Summary
Figure 1 The enterprise data value gap 3. Introduction
Figure 2 Reservoir versus circulating medium 3. Introduction
Figure 3 Intellectual foundations of the Data Swimming Pool framework 4. Background
Figure 4 The correlation tax cycle 5. Current Challenges in Enterprise Data
Figure 5 Six decades of enterprise data architecture 6. Evolution of Enterprise Data Architecture
Figure 6 Capability accretion versus the surviving constraint 6. Evolution of Enterprise Data Architecture
Figure 7 The four structural deficits and their operational consequences 7. Why Another Architecture?
Figure 8 The Data Swimming Pool conceptual model 8. Introducing Data Swimming Pool
Figure 9 The ten core design principles, grouped into five thematic clusters 9. Core Design Principles
Figure 10 Data Swimming Pool layered reference architecture 10. System Architecture
Figure 11 Component interaction topology, separating the continuous write path from the on-demand read path 10. System Architecture
Figure 12 Reference deployment topology 10. System Architecture
Figure 13 Streaming layer architecture 11. Streaming Layer
Figure 14 Event-time processing with watermarks, allowed lateness, and correlation revision 11. Streaming Layer
Figure 15 Batch layer architecture 12. Batch Layer
Figure 16 The reconciliation protocol 12. Batch Layer
Figure 17 Data ingestion workflow 13. Data Ingestion Layer
Figure 18 Quarantine and remediation flow 13. Data Ingestion Layer
Figure 19 Correlation Engine internal architecture 14. Correlation Engine
Figure 20 Confidence scoring pipeline, illustrated for a single correlation 14. Correlation Engine
Figure 21 Correlation lifecycle state machine 14. Correlation Engine
Figure 22 Rules engine architecture 15. Rules Engine
Figure 23 AI Insight Engine architecture 16. AI Insight Engine
Figure 24 Correlation-grounded retrieval sequence 16. AI Insight Engine
Figure 25 Enterprise knowledge graph schema fragment 17. Knowledge Graph Layer
Figure 26 The metadata plane 18. Metadata and Data Lineage
Figure 27 End-to-end lineage for a single derived insight, illustrative 18. Metadata and Data Lineage
Figure 28 Canonical event lifecycle 19. Event Relationships
Figure 29 Event relationship taxonomy 19. Event Relationships
Figure 30 The Autonomy Ladder 20. Autonomous Decision Support
Figure 31 Autonomy assignment by reversibility and confidence 20. Autonomous Decision Support
Figure 32 Autonomy Controller decision flow for a proposed L4 action 20. Autonomous Decision Support
Figure 33 Alert generation and suppression pipeline 21. Alert Generation
Figure 34 Enterprise integration architecture 22. Enterprise Integration
Figure 35 Zero-trust security architecture 23. Security Architecture
Figure 36 Privacy-by-design data flow with purpose binding 24. Privacy
Figure 37 Federated governance operating model 25. Governance
Figure 38 Four-level partitioning strategy 26. Scalability
Figure 39 Multi-region high-availability topology 27. High Availability
Figure 40 Illustrative latency budget for the hot correlation path 28. Performance Considerations
Figure 41 Banking correlation architecture (illustrative) 29. Banking Use Case
Figure 42 Banking event sequence (illustrative) 29. Banking Use Case
Figure 43 Healthcare correlation architecture (illustrative) 30. Healthcare Use Case
Figure 44 Healthcare deterioration correlation workflow (illustrative) 30. Healthcare Use Case
Figure 45 Manufacturing predictive quality and maintenance workflow (illustrative) 31. Manufacturing Use Case
Figure 46 Retail unified commerce correlation workflow (illustrative) 32. Retail Use Case
Figure 47 Smart city multi-agency correlation workflow (illustrative) 33. Smart Cities Use Case
Figure 48 Cybersecurity cross-telemetry threat correlation workflow (illustrative) 34. Cybersecurity Use Case
Figure 49 Government inter-agency service correlation workflow (illustrative) 35. Government Use Case
Figure 50 Telecommunications network and experience correlation workflow (illustrative) 36. Telecommunications Use Case
Figure 51 Insurance correlation pattern (illustrative) 36A. Insurance, Logistics, Energy and Aviation
Figure 52 Energy correlation pattern (illustrative) 36A. Insurance, Logistics, Energy and Aviation
Figure 53 Capability profile comparison 37. Comparison with Existing Technologies
Figure 54 Benefit taxonomy across five categories 38. Benefits
Figure 55 Illustrative benefit realization curve 38. Benefits
Figure 56 Limitations positioned by tractability and severity 39. Challenges and Limitations
Figure 57 Research agenda across four horizons 40. Future Roadmap
Figure 58 Phased adoption with explicit exit gates 40. Future Roadmap
Figure 59 Separation of adopted foundations from proposed contributions 41. Conclusion
Figure 60 Reference deployment topology Appendix B — Reference Implementation Blueprint
Figure 61 Correlation maturity model Appendix D — Adoption Maturity Assessment

Tables #

# Caption Chapter
Table 1 Capability comparison across enterprise data architecture paradigms 1. Executive Summary
Table 2 Data Swimming Pool’s Correlation Engine relative to classical complex event processing 4. Background
Table 3 Symptoms and structural causes of enterprise data underperformance 5. Current Challenges in Enterprise Data
Table 4 Windows of actionability versus typical detection latency 5. Current Challenges in Enterprise Data
Table 5 Generational comparison of enterprise data architectures 6. Evolution of Enterprise Data Architecture
Table 6 Tests applied to justify a new architectural paradigm 7. Why Another Architecture?
Table 7 Assessment of extending existing paradigms 7. Why Another Architecture?
Table 8 Technical preconditions for industrialized correlation 7. Why Another Architecture?
Table 9 The swimming pool metaphor mapped to architectural commitments 8. Introducing Data Swimming Pool
Table 10 Data Lake versus Data Swimming Pool 8. Introducing Data Swimming Pool
Table 11 The ten core design principles with their architectural realizations and violation consequences 9. Core Design Principles
Table 12 Layer responsibilities, prohibitions, and candidate technologies 10. System Architecture
Table 13 Delivery semantics and their correlation implications 11. Streaming Layer
Table 14 Correlation state categories 11. Streaming Layer
Table 15 Streaming layer technology selection matrix 11. Streaming Layer
Table 16 Batch workload classes 12. Batch Layer
Table 17 Open table format properties and their correlation applications 12. Batch Layer
Table 18 Cost and suitability by correlation characteristic 12. Batch Layer
Table 19 Ingestion patterns by source type 13. Data Ingestion Layer
Table 20 Classification tags applied at ingestion 13. Data Ingestion Layer
Table 21 Ingestion observability metrics 13. Data Ingestion Layer
Table 22 Correlation modalities compared 14. Correlation Engine
Table 23 Correlation scope declaration elements 14. Correlation Engine
Table 24 Correlation Engine failure modes 14. Correlation Engine
Table 25 Correlation Engine relative to classical CEP 14. Correlation Engine
Table 26 Rule classes, override permissions, and typical execution tiers 15. Rules Engine
Table 27 Rules engine observability 15. Rules Engine
Table 28 AI model portfolio 16. AI Insight Engine
Table 29 Required constraints on AI agents 16. AI Insight Engine
Table 30 Insight Object essential fields 16. AI Insight Engine
Table 31 Graph-native analytics and their correlation applications 17. Knowledge Graph Layer
Table 32 Edge lifecycle mechanisms 17. Knowledge Graph Layer
Table 33 Graph technology selection considerations 17. Knowledge Graph Layer
Table 34 Catalog graph versus living knowledge graph 17. Knowledge Graph Layer
Table 35 Metadata classes maintained by the plane 18. Metadata and Data Lineage
Table 36 Five levels of lineage depth 18. Metadata and Data Lineage
Table 37 Active metadata behaviours 18. Metadata and Data Lineage
Table 38 Relationship types with detection modalities and confidence ceilings 19. Event Relationships
Table 39 Relationship composition structures 19. Event Relationships
Table 40 Autonomy Ladder level specifications 20. Autonomous Decision Support
Table 41 Autonomy failure controls 20. Autonomous Decision Support
Table 42 Alert clustering mechanisms 21. Alert Generation
Table 43 Alert quality metrics 21. Alert Generation
Table 44 Relationship to existing platform investments 22. Enterprise Integration
Table 45 Agent tool surface 22. Enterprise Integration
Table 46 Security controls mapped to architectural layers 23. Security Architecture
Table 47 Insider risk controls 23. Security Architecture
Table 48 Privacy techniques and their applicability 24. Privacy
Table 49 Data subject rights and framework capability 24. Privacy
Table 50 Correlation scope lifecycle 25. Governance
Table 51 Governance questions applied to correlation types 25. Governance
Table 52 Regulatory alignment at a conceptual level 25. Governance
Table 53 Scaling dimensions and strategies 26. Scalability
Table 54 Elasticity by component 26. Scalability
Table 55 Estimated practical scale boundaries 26. Scalability
Table 56 Four availability types 27. High Availability
Table 57 Availability tiers with recovery objectives 27. High Availability
Table 58 Failure scenarios and their correlation integrity impact 27. High Availability
Table 59 Operational availability practices 27. High Availability
Table 60 Proposed service level objectives by path 28. Performance Considerations
Table 61 Throughput characteristics 28. Performance Considerations
Table 62 Banking correlation scopes (illustrative) 29. Banking Use Case
Table 63 Banking capability comparison (illustrative) 29. Banking Use Case
Table 64 Illustrative outcome directions 29. Banking Use Case
Table 65 Healthcare correlation scopes (illustrative) 30. Healthcare Use Case
Table 66 Illustrative outcome directions for manufacturing 31. Manufacturing Use Case
Table 67 Illustrative outcome directions for retail 32. Retail Use Case
Table 68 Illustrative outcome directions for smart cities 33. Smart Cities Use Case
Table 69 Cybersecurity correlation scopes (illustrative) 34. Cybersecurity Use Case
Table 70 Illustrative outcome directions for cybersecurity 34. Cybersecurity Use Case
Table 71 Autonomy constraints for government decision types 35. Government Use Case
Table 72 Illustrative outcome directions for telecommunications 36. Telecommunications Use Case
Table 73 Logistics correlation scopes (illustrative) 36A. Insurance, Logistics, Energy and Aviation
Table 74 Aviation correlation scopes (illustrative) 36A. Insurance, Logistics, Energy and Aviation
Table 75 Cross-sector correlation pattern synthesis 36A. Insurance, Logistics, Energy and Aviation
Table 76 Master architectural comparison matrix 37. Comparison with Existing Technologies
Table 77 Detailed feature matrix 37. Comparison with Existing Technologies
Table 78 Total cost of ownership considerations 37. Comparison with Existing Technologies
Table 79 Selection guidance 37. Comparison with Existing Technologies
Table 80 Fifteen claimed benefits with mechanisms and falsification conditions 38. Benefits
Table 81 Benefit measurement approaches 38. Benefits
Table 82 Twelve tractable challenges with mitigations and residual risk 39. Challenges and Limitations
Table 83 Architecture-specific failure modes 39. Challenges and Limitations
Table 84 Priority open problems 40. Future Roadmap
Table 85 Adoption phases with objectives, deliverables, and gates 40. Future Roadmap
Table 86 Adoption anti-patterns 40. Future Roadmap
Table 87 External developments that would materially improve the framework’s viability 40. Future Roadmap
Table 88 Component selection by layer Appendix B — Reference Implementation Blueprint
Table 89 Indicative sizing profiles Appendix B — Reference Implementation Blueprint
Table 90 Indicative build sequence in two-week sprints Appendix B — Reference Implementation Blueprint
Table 91 Minimum viable correlation platform Appendix B — Reference Implementation Blueprint
Table 92 Correlation type reference with confidence ceilings and default lifetimes Appendix C — Canonical Event and Correlation Schemas
Table 93 Assessment instrument across six dimensions Appendix D — Adoption Maturity Assessment
Table 94 Score interpretation Appendix D — Adoption Maturity Assessment
Table 95 Common assessment profiles Appendix D — Adoption Maturity Assessment

Listings #

# Caption Chapter
Listing C-1 Canonical event schema Appendix C — Canonical Event and Correlation Schemas
Listing C-2 Correlation schema Appendix C — Canonical Event and Correlation Schemas
Listing C-3 Insight Object schema Appendix C — Canonical Event and Correlation Schemas
Listing C-4 Correlation scope declaration Appendix C — Canonical Event and Correlation Schemas

Totals: 63 figures · 95 tables · 4 schema listings.


Next: 1. Executive Summary →