Data Lineage Tracker

Implement end-to-end data lineage tracking using OpenLineage, DataHub, and Marquez for impact analysis, debugging, and compliance.

Activation Triggers

Activate on: "data lineage", "OpenLineage", "DataHub", "Marquez", "impact analysis", "data catalog", "column lineage", "data discovery", "data provenance", "downstream impact"

NOT for: Data quality checks → data-quality-guardian | dbt model documentation → dbt-analytics-engineer | Schema evolution → schema-evolution-manager

Quick Start

1. Choose platform — DataHub (full catalog + lineage), Marquez (lineage-focused), Amundsen, OpenMetadata
2. Instrument pipelines — integrate OpenLineage with Airflow, Spark, and dbt for automatic lineage
3. Capture column-level lineage — trace which source columns feed which target columns
4. Build impact analysis — before changing a table, see all downstream consumers
5. Integrate with incidents — link data quality alerts to lineage for root cause analysis

Core Capabilities

Domain	Technologies
Lineage Standard	OpenLineage 1.x (open standard, LFAI)
Catalogs	DataHub 0.14+, OpenMetadata 1.5+, Amundsen
Lineage Backend	Marquez (OpenLineage reference), DataHub lineage
Integrations	Airflow OpenLineage, Spark OpenLineage, dbt, Great Expectations
Visualization	DataHub lineage graph, Marquez UI, dbt docs

Architecture Patterns

OpenLineage Event Flow

Airflow Task              Spark Job                 dbt Model
     │                         │                         │
     ├─ START event ──→        ├─ START event ──→        ├─ START event ──→
     │                         │                         │
     ├─ RUNNING event ─→       ├─ RUNNING event ─→       │
     │                         │                         │
     ├─ COMPLETE event ─→      ├─ COMPLETE event ─→      ├─ COMPLETE event ─→
     │  (with I/O datasets)    │  (with I/O datasets)    │  (with I/O datasets)
     ↓                         ↓                         ↓
              ┌────────────────────────────────────┐
              │     OpenLineage Backend            │
              │  (Marquez / DataHub / custom)      │
              │                                    │
              │  Stores: job runs, datasets,       │
              │  input/output relationships,       │
              │  column-level lineage              │
              └────────────────────────────────────┘

Impact Analysis Query

# DataHub GraphQL: find all downstream dependencies of a dataset
query = """
{
  dataset(urn: "urn:li:dataset:(urn:li:dataPlatform:snowflake,prod.fct_orders,PROD)") {
    downstream: relationships(
      input: { types: ["DownstreamOf"], direction: INCOMING, count: 50 }
    ) {
      relationships {
        entity {
          urn
          ... on Dataset {
            name
            platform { name }
            properties { description }
          }
        }
      }
    }
  }
}
"""

# Result: all dashboards, models, and exports that depend on fct_orders
# Use this BEFORE making schema changes

Column-Level Lineage

Source Tables                  Transformation              Target Table
─────────────                 ──────────────              ────────────
raw_payments.amount    ──→    SUM(amount)           ──→  fct_revenue.total_revenue
raw_payments.currency  ──→    exchange_rate_convert  ──→  fct_revenue.total_revenue_usd
raw_orders.order_id    ──→    JOIN key              ──→  fct_revenue.order_id
raw_customers.name     ──→    COALESCE(name, email) ──→  fct_revenue.customer_name

Column-level lineage answers:
  "Where does fct_revenue.total_revenue_usd come from?"
  → raw_payments.amount + raw_payments.currency via exchange_rate_convert

Anti-Patterns

1. Manual lineage documentation — manually maintained lineage diagrams go stale immediately; automate via OpenLineage
2. Table-level only — table lineage is not enough; column-level lineage is required for meaningful impact analysis
3. No integration with incidents — lineage is most valuable during incidents; link quality alerts to lineage graph
4. Lineage without ownership — every dataset in the catalog must have an owner team for accountability
5. Collecting without using — lineage data is useless if nobody queries it; embed impact analysis in CI and change management

Quality Checklist

• OpenLineage integrated with Airflow, Spark, and dbt
• Column-level lineage captured (not just table-level)
• All datasets have assigned owners in the catalog
• Impact analysis run before schema changes (CI check)
• Lineage graph accessible via UI (DataHub, Marquez)
• Data quality alerts link to lineage for root cause tracing
• Cross-platform lineage works (e.g., Kafka → Spark → Snowflake)
• Lineage metadata refreshed on every pipeline run (not periodic batch)
• Search and discovery enabled: analysts can find datasets by keyword
• Retention policy on lineage events (keep 90 days of run history)