Data Mesh: Domain-Oriented Data Ownership, Data Products, and Self-Serve Data Infrastructure

The central data team model breaks down at scale. A single team responsible for all data pipelines becomes a bottleneck: domain teams wait weeks for their data to be onboarded, pipelines break because the central team doesn't understand the source domain, and the data lake becomes a data swamp nobody trusts.

Data mesh is the organizational and architectural response. Like microservices distributed system ownership to product teams, data mesh distributes data ownership to domain teams — while providing a platform that makes producing and consuming data products self-service.

---

The Four Principles

1. Domain-oriented decentralized data ownership Each domain team owns the data they produce — including making it available to other consumers.

2. Data as a product Domain teams treat their data outputs as products: with defined schemas, SLAs, documentation, and quality metrics. Data consumers are the customers.

3. Self-serve data infrastructure platform A platform team provides the infrastructure that makes it easy for domains to publish and consume data products without deep data engineering expertise.

4. Federated computational governance Global standards (data classification, lineage, privacy requirements) are enforced automatically, not through central gatekeeping.

---

Data Mesh vs Data Lake

---

What a Data Product Is

A data product is a dataset treated as a software product:

## Data Product: Orders — Daily Summary

Owner: Payments Team
Domain: Orders

### Contract
- Schema: orders_daily (see schema below)
- Freshness SLA: Data available by 8:00 AM UTC
- Quality SLA: < 0.1% null order_id; all amounts > 0
- Retention: 2 years

### Access
- Location: s3://data-products/orders/daily/year={year}/month={month}/day={day}/
- Format: Parquet (snappy compressed)
- Catalog: orders.daily in Apache Atlas / DataHub
- Request access: data-mesh-access@yourcompany.com

### Schema
| Column | Type | Description |
|---|---|---|
| order_id | UUID | Unique order identifier |
| tenant_id | UUID | Tenant who placed the order |
| status | STRING | PENDING, CONFIRMED, SHIPPED, DELIVERED, CANCELLED |
| total_cents | INT64 | Order total in cents |
| item_count | INT32 | Number of line items |
| created_date | DATE | Order creation date |

### Changelog
v2.0 (2026-06-01): Added item_count column
v1.0 (2025-01-01): Initial release

---

Building a Data Product Pipeline

# pipelines/orders/daily_summary.py
# Domain team owns and maintains this pipeline

from datetime import date, timedelta
import duckdb
import boto3
import json

def build_orders_daily_summary(target_date: date) -> None:
    """Build the orders daily summary data product."""

    # Read from the domain's operational database
    conn = duckdb.connect()
    conn.execute("INSTALL postgres; LOAD postgres;")
    conn.execute(f"ATTACH 'dbname=orders host=orders-db.internal' AS orders_db (TYPE postgres)")

    # Transform to the data product schema
    result = conn.execute(f"""
        SELECT
            o.id AS order_id,
            o.tenant_id,
            o.status,
            o.total_cents,
            COUNT(oi.id) AS item_count,
            CAST(o.created_at AS DATE) AS created_date
        FROM orders_db.orders o
        JOIN orders_db.order_items oi ON oi.order_id = o.id
        WHERE CAST(o.created_at AS DATE) = '{target_date}'
        GROUP BY 1, 2, 3, 4, 6
    """)

    # Convert to Arrow for Parquet writing
    df = result.arrow()

    # Write to S3 with Hive partitioning
    s3_path = (
        f"s3://data-products/orders/daily/"
        f"year={target_date.year}/"
        f"month={str(target_date.month).zfill(2)}/"
        f"day={str(target_date.day).zfill(2)}/"
        f"data.parquet"
    )

    import pyarrow.parquet as pq
    import pyarrow.fs as pafs

    s3fs = pafs.S3FileSystem(region='us-east-1')
    pq.write_table(df, s3_path, filesystem=s3fs, compression='snappy')

    # Register in data catalog (DataHub / Glue)
    register_data_product(
        product_id='orders.daily',
        partition=target_date,
        row_count=len(df),
        s3_path=s3_path,
    )

    # Run quality checks (fail and alert if violated)
    run_quality_checks(df, product_id='orders.daily', date=target_date)


def run_quality_checks(df, product_id: str, date: date) -> None:
    """Validate data product quality SLAs."""
    import pyarrow.compute as pc

    checks = {
        'no_null_order_ids': pc.sum(pc.is_null(df['order_id'])).as_py() == 0,
        'positive_totals': pc.min(df['total_cents']).as_py() >= 0,
        'row_count_reasonable': len(df) > 0,
    }

    failures = [check for check, passed in checks.items() if not passed]

    if failures:
        # Alert data product owner
        send_alert(
            to='payments-team@yourcompany.com',
            subject=f'Data product quality check failed: {product_id} {date}',
            body=f'Failed checks: {", ".join(failures)}',
        )
        raise ValueError(f'Quality checks failed: {failures}')

---

Self-Serve Data Platform: What It Provides

Data Platform Capabilities

For Data Producers (Domain Teams)

• Pipeline templates (dbt, Airflow DAG, AWS Glue) — copy and customize
• Schema registry — validate and version schemas
• Data catalog registration — automatic from pipeline metadata
• Quality check framework — declare rules, platform runs them
• S3 path conventions + IAM role provisioning (Crossplane)
• Parquet/Delta writer SDK — consistent format without expertise

For Data Consumers (Analysts, Other Teams)

• Data catalog with search (DataHub, Apache Atlas)
• Access request workflow (request → auto-provisioned IAM)
• Query engine (Athena, Trino) with pre-configured connections
• Data lineage — see what upstream products feed this one
• Freshness monitoring — see last-updated time for each product
• Sample data in catalog — see a few rows before requesting access

---

## Federated Governance: Global Standards

```yaml
# data-product-spec.yaml — schema enforced by platform
apiVersion: data.platform.yourcompany.com/v1
kind: DataProduct
metadata:
  name: orders-daily
  owner: payments-team@yourcompany.com
  domain: payments
spec:
  classification: INTERNAL     # PUBLIC | INTERNAL | CONFIDENTIAL | RESTRICTED
  containsPII: false           # If true: platform auto-enforces column-level encryption
  retentionDays: 730           # Platform auto-expires after this
  freshnessTarget:
    type: daily
    by: "08:00 UTC"
  qualityRules:
    - column: order_id
      rule: NOT_NULL
    - column: total_cents
      rule: POSITIVE
  outputPorts:
    - type: s3
      path: s3://data-products/orders/daily/
      format: parquet
    - type: sql
      database: prod_analytics
      table: orders.daily

The platform enforces the spec: if containsPII: true, columns tagged as PII are automatically encrypted and access is logged. Domain teams don't need to implement this themselves.

---

Migration Path: Lake → Mesh

## Migration Strategy: Centralized Lake to Data Mesh

Phase 1: Identify domains and data products (2–4 weeks)
- Map all datasets in current lake to owning domains
- Identify top 5 high-value, high-use datasets
- Define data product contracts for those 5

Phase 2: Platform foundation (4–8 weeks)
- Set up data catalog (DataHub or Apache Atlas)
- Standardize storage (S3 + Parquet + Hive partitioning)
- Create pipeline template for domain teams
- Set up access management (IAM roles via Crossplane)

Phase 3: Pilot migration (4–6 weeks)
- Migrate 2–3 high-value datasets with pilot domain teams
- Teams take ownership, write quality checks, publish contracts
- Central team supports but doesn't own

Phase 4: Scale (ongoing)
- Each quarter: onboard 3–5 new domain teams
- Sunset central pipelines as domains take ownership
- Measure: adoption rate, time-to-publish, query SLA adherence

---

Working With Viprasol

We design and implement data mesh architectures — domain boundary identification, data product design, platform infrastructure (DataHub, Apache Atlas, S3 + Parquet), and the governance layer that keeps data trustworthy at scale.

→ Talk to our team about data architecture and analytics infrastructure.

---

See Also

• Data Engineering Pipeline — ELT pipeline patterns
• Embedded Analytics — serving data products to customers
• Product Analytics — event data as a data product
• Cloud Solutions — data infrastructure and analytics
• AI/ML Services — ML models consuming data mesh products