48
Certified Data Products
Dashboard
admin@ulticare.org
Botswana
Data Platform Dashboard
Certified data products, lineage, governance & control plane
17
Domains
30
Iceberg Tables
99.9%
SLA Uptime
Data Flow Check
Verify that the operational data plane (Trino) is reachable.
Not run
Data Flow
Source Systems
Debezium CDC
Kafka
Spark
Iceberg
Trino
DataHub
Control Plane
Consumers
Quick Access
External tools & consoles
Data Platform Architecture
Layered design, runtime components, contracts, governance and known gaps
Status legend:
Deployed
Configured
Target
Gap
Full design:
docs/data-platform-architecture.md · Topology: docker-compose.dataplatform.yml
Layered Architecture (L0 → L7)
End-to-End Data Flow
Source DBSystem of record
DebeziumWAL / pgoutput
ApicurioAvro contract
Kafka7d · RF3
AirflowHourly DAG
SparkParse + merge
IcebergSnapshot commit
MinIOParquet + manifest
TrinoFederated SQL
Control PlaneRBAC · mask · tenant
ConsumersPortals · APIs · ML
Runtime Components
CDC Connectors
Lakehouse Tables
Orchestration, Contracts & SLOs
Security, Privacy & Tenancy
- Identity: Keycloak 24 (OIDC/SSO), realm
ulticare; JWT carriestenant_countryand roles. - Authorization: Trino file-based access control; admin all, others read-only on
iceberg. - Tenant isolation: row filter
tenant_country = current_user_tenant(); cross-border access requires consent. - PII masking: email, phone and national ID masked for non-admin roles; minimum-necessary filtering by purpose.
- Compliance: POPIA, HIPAA, GDPR, HPCSA rules; consent lifecycle; audit logging with local fallback.
- Contracts: Apicurio Avro (
BACKWARD), Pact consumer contracts, event JSON Schemas incontracts/events/. - Entitlements: marketplace tiers per product in
data-products/marketplace_entitlements.yaml. - Audit: Trino query history, access logs, AI-inference audit events, lineage evidence.
Implementation Status & Gaps
Source Documents & Artifacts
Certified Data Products
48 domain-aligned & stakeholder-specific products with schema, SLAs & access control
Data Lineage
End-to-end traceability from source systems to consumers
Source SystemsPostgres, PostGIS, FHIR
Debezium CDCChange data capture
KafkaEvent streaming
SparkETL orchestration
IcebergLakehouse tables
TrinoQuery engine
DataHubCatalog & lineage
Control PlaneRBAC, masking, audit
ConsumersPortals, APIs, dashboards
Lineage Artifacts
Lineage Coverage
100%Domain tables
100%Stakeholder products
100%CDC connectors
100%ETL pipelines
Governance & Catalog
Data contracts, semantic models, KPI marts & marketplace entitlements
Catalog Assets
Product Catalog
data-products/product_catalog.json
data-products/product_catalog.json
Governance Catalogue
datahub/config/governance-catalogue.yml
datahub/config/governance-catalogue.yml
Marketplace Entitlements
data-products/marketplace_entitlements.yaml
data-products/marketplace_entitlements.yaml
Semantic Models
data-products/semantic_models.yaml
data-products/semantic_models.yaml
KPI Marts
data-products/kpi_marts.yaml
data-products/kpi_marts.yaml
Feature Views
data-products/feature_views.yaml
data-products/feature_views.yaml
Data Contracts
- Event Contracts: contracts/events/ (17 domains)
- Trino RBAC: trino/config/rules.json, access-control.json
- Column Masking: trino/config/column-masking.json
- Data Quality: Airflow DAG quality gates
- SLA: Hourly/daily freshness per product
Consumers & Entitlements
Data product consumers and marketplace entitlements
Product Consumers & Entitlements
| Product | Tier | Entitled roles | Consumers |
|---|
Pipeline & Data Quality
CDC to ETL triggers, data quality gates, and contract validation
CDC to ETL Trigger Flow
Source DB ChangeDebezium CDC
Kafka TopicEvent published
Airflow DAG TriggerETL job starts
Iceberg TableNessie commit
Trino QueryData available
Data Quality Gates
- Freshness: hourly (operational) / daily (analytical)
- Completeness: >99% per critical table
- Accuracy: schema validation on ingest (Apicurio)
- Contract validation: Pact + JSON Schema
- SLA monitoring via Airflow + DataHub
Not run
Per-Epic Sequence Diagrams
Flows are rendered from the .mmd sources in docs/diagrams/ using the vendored Mermaid build — no external CDN. Each epic shows the real participants, topics and tables from this deployment.
Diagrams render when this tab is opened.
Live Tooling
Dashboards
Live KPIs and distributions queried from the Iceberg lakehouse through Trino
Not loaded yet — open this tab to query Trino.
Live SQL over iceberg.ulticare. Only CDC-mirrored tables are populated; clinical topics (patient_risk, vitals, chw_visits) stay empty until those connectors are mirrored.
Prescription Review Status
pharmacy__prescriptionsLoading…
Drone Delivery Pipeline
mobile_clinic__drone_deliveriesLoading…
Top Prescribed Medications
pharmacy__prescriptionsLoading…
Stock Below Reorder Point
pharmacy__inventoryLoading…
Prescription Volume by Day
pharmacy__prescriptionsLoading…
SDG Analytics
Sustainable Development Goal indicators measured from the Iceberg lakehouse through Trino
Not loaded yet — open this tab to query Trino.
Scores are computed live from the CDC-mirrored marts (household__*, pharmacy__*, mobile_clinic__*). The alignment index averages only goals whose source fields are populated; goals marked Data gap stay visible so missing pipelines are explicit — SDG-16 needs the compliance/audit tables mirrored first.
Last-Mile Mode Share
mobile_clinic__drone_deliveriesLoading…
ESG Analytics
Environmental, social and governance signals measured from the Iceberg lakehouse through Trino
Not loaded yet - open this tab to query Trino.
E and S are computed live from the CDC-mirrored marts (household__*, pharmacy__*, mobile_clinic__*). Governance stays a Data gap until the compliance/consent/audit tables are mirrored. Household utility fields are sparse in the seed, so water, sanitation and electricity read Data gap instead of a misleading 0%.
Pillar Scores
mean of measured indicators per pillarLoading...
Business Analytics
Commercial pipeline, campaign coverage and donor capital read from the business bronze marts
Not loaded yet — open this tab to query Trino.
Sales-engine, CRM and donors persist to SQLite, so Debezium cannot capture them. business-extract snapshots their APIs into Kafka and the hourly etl_business_analytics DAG MERGEs the records into business__* Iceberg mirrors — the portal reads them through the same Trino path as every other mart. All monetary values are USD.
Pipeline Funnel
business__sales_opportunitiesLoading…
Pipeline Mix by Sector
open pipeline valueLoading…
Forecast by Close Month
open pipeline, weighted forecast and won valueLoading…
Value Concentration
deal value by sector and stageLoading…
Donor Capital
budget vs disbursed by programLoading…
Country Allocation
business__donor_country_allocationsLoading…
Campaign Coverage
people reached vs coverage targetLoading…
Data Control Plane
Ownership, classification, quality, lifecycle, audit & access reviews
Ownership & Stewardship
17 product owners across 17 business domains, each accountable for definition, quality and lifecycle.
data-governance
moh-analytics
logistics-data
pharmacy-analytics
env-health-analytics
moh-surveillance
health-economics
mental-health-analytics
social-determinants-analytics
chronic-disease-analytics
nutrition-analytics
moh-executive
donor-relations
clinical-analytics
chw-program
pharmacy-ops
lab-ops
citizen-analytics
nurse-program
telehealth-analytics
referral-analytics
nhi-analytics
supplychain-analytics
workforce-analytics
facility-analytics
compliance-analytics
quality-analytics
finance-analytics
wearables-analytics
fleet-analytics
chw-telemetry-analytics
immunization-analytics
maternal-analytics
oral-health-analytics
mental-health-analytics
emergency-analytics
home-health-analytics
social-work-analytics
caregiver-analytics
specialist-analytics
rehab-analytics
radiology-analytics
telco-analytics
wellness-analytics
it-ops-analytics
- Domain stewards assigned per business capability
- Stewardship dashboard in DataHub
Classification, PII & Residency
- Trino column masking
trino/config/column-masking.json - Row-level security via
access-control.json - Tenant isolation via multi-tenant IAM
- Cross-border data flow controls
Quality SLOs
- Freshness: hourly (operational) / daily (analytical)
- Completeness: >99% per critical table
- Accuracy: schema validation on ingest
- DataHub freshness monitoring enabled
Lifecycle Gates
- Airflow orchestration with retries & SLA alerts
- Debezium CDC with exactly-once semantics
- Iceberg time-travel & rollback capability
- Schema evolution with compatibility checks
Audit Evidence
- Lineage:
docs/lineage/*, DataHub lineage API - Event contracts:
contracts/events/ - Trino query history & access logs
- CDC offset tracking
Access Reviews & Adoption
- Marketplace entitlements per product
data-products/marketplace_entitlements.yaml - RBAC enforced in Trino & DataHub
- Adoption metrics via DataHub usage analytics
- Quarterly access review workflow