Data Streaming

Modern Best Practices: choose the event backbone and stream processor separately, treat schemas and replay as product interfaces, default to event-time processing for stateful analytics, and verify managed-service behavior from primary docs before making vendor-specific recommendations.

Primary sources live in data/sources.json. Refresh time-sensitive claims against official docs before making definitive recommendations about managed services, version-specific features, limits, or pricing.

This skill covers the data platform side of streaming: event backbones, CDC, stateful processing, schema governance, and real-time delivery into lakes, warehouses, search, or serving systems.

When to Use

• Choose between Kafka, Redpanda, Pulsar, Kinesis, or managed Kafka offerings
• Design topic strategy, partitioning, retention, replay, and ordering guarantees
• Build or fix CDC pipelines with Debezium, Flink CDC, or managed database-streaming tools
• Choose between Flink, Kafka Streams, Spark Structured Streaming, or lighter transformation paths
• Define schema registry, compatibility, contract, and tombstone handling rules
• Deliver streams into Iceberg, Hudi, Delta, ClickHouse, warehouses, caches, or search systems
• Review streaming SLOs, lag, checkpointing, reprocessing, and operational failure modes
• Plan Kafka broker upgrades, the ZooKeeper-to-KRaft migration, consumer-protocol (KIP-848) or share-group (KIP-932) adoption, and client producer defaults

When NOT to Use

• Lakehouse storage formats, catalogs, or medallion architecture -> Use data-lake-platform
• OLTP schema tuning or transactional query optimization -> Use data-sql-optimization
• Event-driven application architecture, CQRS, or domain event design -> Use software-architecture-design
• BI dashboard automation and Metabase APIs -> Use data-metabase
• Product instrumentation and attribution strategy -> Use marketing-product-analytics

Triage Questions

1. What is the real requirement: operational events, CDC, analytical enrichment, or customer-facing low-latency delivery?
2. What matters most: portability, managed simplicity, geo-replication, cost, or end-to-end latency?
3. Where must ordering hold: globally, per key, or only within a local processing step?
4. What is the replay model: full retention, compacted snapshots, time-bounded backfills, or one-shot delivery?
5. Which guarantees are required: at-most-once, at-least-once, or business-level exactly-once with idempotent sinks?
6. Which downstream systems consume the stream: lakehouse tables, warehouses, search, caches, APIs, or ML features?
7. What is the operational baseline: small team, platform team, managed service, or self-hosted multi-region cluster?

Default Workflow (Use Unless User Overrides)

1. Choose the backbone first with references/platform-selection.md.
2. Define topic, key, retention, replay, and schema strategy before discussing processors. Use assets/topic-contract-template.md.
3. Choose the processing model with references/stream-processing-patterns.md: pass-through, enrich, aggregate, join, dedupe, or CDC normalization.
4. Lock CDC and schema-governance rules with references/cdc-and-schema-governance.md and assets/cdc-rollout-checklist.md.
5. Define delivery and sink behavior: upserts, deletes, late data, watermarking, and reprocessing boundaries.
6. Add SLOs, lag monitoring, checkpoint and savepoint policy, and incident drills with references/operations-and-slos.md.
7. Score tradeoffs explicitly with assets/streaming-platform-scorecard.md when the user asks for the "best" platform.

Default Baseline

• Backbone: Kafka-compatible event log unless a clear managed-service or multi-tenant requirement pushes elsewhere
• Stream processing: Flink for stateful event-time pipelines; Kafka Streams for lighter in-app processing
• CDC: log-based CDC first; avoid trigger-based CDC unless constraints force it
• Contracts: registry-backed schemas for shared or long-lived topics
• Reprocessing: plan for replay before launch; do not treat backfills as exceptional
• Sinks: design sink idempotency explicitly; "exactly-once" claims are incomplete without sink behavior

Backbone Decision Table

Requirement	Kafka (self-hosted)	Redpanda	WarpStream / AutoMQ	Pulsar	Kinesis
Broadest connector/CDC ecosystem	best	good	good (Kafka API)	limited	limited
Operational simplicity	medium (KRaft-only in 4.x, no ZooKeeper to run)	good (single binary)	best (serverless or S3-backed)	poor	best
Cost at high throughput	price the workload	price the workload	price storage, requests, and egress	price the workload	price capacity mode and reads
Multi-tenancy / namespace isolation	limited	limited	limited	best	AWS-only
Geo-replication built-in	via MirrorMaker 2	via MirrorMaker 2	limited	native	AWS-only
Kafka API compatibility	canonical	verify required APIs and versions	verify required APIs and versions	partial	no
Queue semantics (share groups)	KIP-932 (check status for your version)	check vendor docs	check vendor docs	native	no
Cloud portability	high	high	medium (S3 dependency)	high	none

Ownership and product lines of managed Kafka-compatible services change through acquisitions. Check the vendor's current docs and support terms before recommending one.

Exactly-Once Decision Path

Need exactly-once?
  -> Is the sink idempotent or transactional?
       No  -> Add sink-level deduplication key or upsert semantics first
       Yes -> Enable broker/processor exactly-once:
                Kafka: enable.idempotence=true + transactional.id
                Flink: CheckpointingMode.EXACTLY_ONCE + Sink V2 committer (two-phase commit)
                Downstream Kafka consumers: isolation.level=read_committed
                  (the default read_uncommitted also reads aborted records)
  -> Does the sink support two-phase commit?
       No  -> Business-level idempotency (dedupe key + conditional write)
       Yes -> End-to-end exactly-once boundary confirmed
  -> Document the exact guarantee boundary — broker, processor, AND sink

Version-Sensitive Facts (Look Up, Do Not Quote)

Release lines, feature status, and patch regressions change faster than this skill. Before naming a version or calling a feature GA:

• Kafka: read the Apache Kafka release announcements for the current stable line, known regressions fixed in patch releases (Kafka Streams state-store issues in particular), and the status of KIP-932 share groups (preview vs GA) for the exact broker and client versions in use.
• Flink: read the release announcement for the target version. Carry over its own qualifiers: a feature it calls experimental stays experimental in your recommendation. Do not call a feature GA (for example Materialized Tables or disaggregated state) unless the release notes say so.
• Kafka upgrades and adoption: migration order, upgrade traps, consumer-protocol and share-group adoption checks, and per-client producer defaults live in references/kafka-upgrades-and-client-defaults.md.
• Iceberg engines: per-engine v3 read, write, and maintenance status lives in references/engine-table-format-state.md as lookup steps; confirm it in each vendor's current docs.

Stable facts that do not need a lookup:

• Kafka 4.0 removed ZooKeeper; every 4.x cluster is KRaft-only. Controller quorum sizing differs from ZooKeeper ensemble sizing, so do not map 1:1. Migrate non-production first and verify the controller quorum before cutting over.
• KIP-932 share groups let several consumers cooperate on one partition with per-record acknowledgment. That gives queue semantics without partition-per-consumer assignment, and it is the one Kafka mode where pooled (M/M/c) sizing applies.
• Flink 2.0 removed the legacy SinkFunction/TwoPhaseCommitSinkFunction and AssignerWith*Watermarks APIs. Use Sink V2 and WatermarkStrategy. For a 1.x-to-2.x upgrade, take a savepoint and test restore with the target job: compatible state, stable operator IDs, supported connectors, and changed APIs determine whether it works. Check the Flink upgrade guide before migrating Table API jobs or relying on SQL Gateway behavior.

Iceberg Streaming Ingestion Pattern

Standard production stack: Kafka -> Flink (Dynamic Iceberg Sink) -> Iceberg table -> compaction job.

# Illustrative starting point, not from the Flink Dynamic Iceberg Sink blog — tune on the real workload
execution.checkpointing.interval = 5 min   # each checkpoint is one commit opportunity
write.target-file-size-bytes = 536870912   # Iceberg table default (512 MB); lowering it creates more small files
write.fanout.enabled = true                # unordered writes across partitions
# Schema compatibility (e.g. FULL_TRANSITIVE) is a schema-registry subject setting, not a sink option
# Compaction: run on cold partitions; skip hot (current) partition

• Every streaming approach produces small files — pair ingestion with a scheduled compaction job.
• With one commit opportunity per checkpoint and all else equal, a 5-minute interval creates one-fifth as many commit opportunities as a 1-minute interval (80% fewer). That arithmetic does not predict file count: partitions, writer parallelism, rollover, traffic shape, and compaction also matter. Measure file count, size distribution, recovery time, and latency on the real workload before tuning.
• Iceberg v3 (ratified spec): deletion vectors, row lineage (_row_id), variant data, default column values, geometry/geography types, nanosecond timestamps, encryption foundations. Do not quote a DML speed-up multiplier for deletion vectors; the spec gives none. Engine support differs per engine and per operation, so check each engine's current read, write, and maintenance support (references/engine-table-format-state.md) before committing a multi-engine stack to v3 features.

Replay Acceptance Gate

A replay plan is complete only when it answers four separate questions:

1. Source position: Which offsets, timestamps, snapshot, or CDC log position define the replay boundary?
2. State reset: Which processor state and checkpoints are restored, discarded, or rebuilt?
3. Sink behavior: Which idempotency key, upsert, transaction, or dedupe window prevents duplicate business effects?
4. Consumer isolation: How are replay records kept from triggering emails, payments, alerts, or other irreversible side effects twice?

Rehearse one bounded replay through the real sink before launch. Pass only if row/event counts reconcile at source and sink, deletes and late events behave as contracted, consumer lag recovers inside the SLO, and a second replay produces the same business state.

Quick Reference

Task	Resource	When to Use
Choose Kafka vs Redpanda vs Pulsar vs Kinesis	references/platform-selection.md	New platform selection or platform migration
Choose Flink vs Kafka Streams vs Spark	references/stream-processing-patterns.md	Stateful processing, joins, windows, or low-latency transforms
Design CDC and schema evolution	references/cdc-and-schema-governance.md	Debezium, snapshots, tombstones, contracts, registry policy
Define lag, replay, failover, and checkpoint policy	references/operations-and-slos.md	Production hardening and incident prevention
Draft topic naming, keys, retention, and schema rules	assets/topic-contract-template.md	New topic or shared event contract
Plan a CDC rollout safely	assets/cdc-rollout-checklist.md	Database-to-stream launch or CDC migration
Compare platform options side by side	assets/streaming-platform-scorecard.md	Decision reviews and recommendation memos

Operating Principles

1. Ordering Is Scoped, Not Global

• Promise ordering only where the platform can really preserve it, usually per partition and key.
• If the business process needs entity-level sequencing, make the key choice explicit.

2. Schemas Are Contracts

• Shared topics need governed evolution rules, owners, compatibility mode, and deprecation windows.
• Plain JSON is acceptable for prototyping, not for durable shared interfaces.

3. Replay Is A First-Class Operation

• Retention, compaction, checkpoints, and sink idempotency define whether replay is safe.
• Do not ship a pipeline that cannot be re-run after bad code or bad data.

4. "Exactly-Once" Is End-To-End, Not A Checkbox

• Broker or processor guarantees are insufficient if the sink can duplicate writes or mishandle deletes.
• State the exact boundary where deduplication or transactional guarantees end.

5. CDC Needs Delete And Snapshot Strategy

• Decide how snapshots, schema changes, tombstones, and source failover behave before launch.
• Downstream consumers must know whether deletes arrive as tombstones, hard deletes, or soft-delete flags.

Templates

• assets/topic-contract-template.md
• assets/cdc-rollout-checklist.md
• assets/streaming-platform-scorecard.md

Known Traps

• Designing the event backbone around broker features before defining domain ownership, event contracts, and replay expectations.
• Treating topic retention as a substitute for a durable system of record, replay plan, or downstream recovery workflow.
• Claiming exactly-once behavior without specifying the guarantee boundary across broker, processor, sink, and side effects.
• Mixing operational events, analytical CDC, and integration commands into the same topics without independent retention, schema, and consumer-SLA rules.
• Shipping CDC streams without idempotency keys, snapshot semantics, tombstone handling, and late-arrival rules agreed by consumers.
• Scaling partitions, consumer groups, and stateful processors independently and then discovering the keying model breaks ordering or hotspot behavior.

Common Anti-Patterns

• Using the stream platform as a generic dumping ground for every event rather than curating contracts by domain and use case.
• Putting business-critical enrichment or policy decisions in opaque stream jobs with no replay procedure, lineage, or owner.
• Letting producers evolve schemas opportunistically while expecting consumers to absorb breaking changes.
• Building low-latency pipelines on top of unstable event keys, non-deterministic joins, or external side-effect calls inside hot-path processors.
• Choosing real-time processing because it sounds strategic when batch or micro-batch would meet the product and cost requirements.
• Treating DLQs as the main error-handling strategy instead of fixing classifier logic, validation, backpressure, and recovery paths upstream.

Navigation

• references/platform-selection.md — Load when choosing between Kafka, Redpanda, Pulsar, Kinesis, or managed offerings; includes decision axes and recommendation protocol.
• references/stream-processing-patterns.md — Load when designing topology, windows, joins, deduplication, or delivery semantics.
• references/cdc-and-schema-governance.md — Load when building CDC pipelines, handling deletes/tombstones, or setting schema registry policy.
• references/operations-and-slos.md — Load when defining SLOs, dashboards, incident patterns, or consumer commit/DLQ policy.
• references/engine-table-format-state.md — Load when reasoning about Iceberg v3, Paimon, or Kafka KIP-1150 diskless status.
• references/control-theory-applied.md — Load when designing lag-aware autoscalers, producer flow control, or watermark tuning.
• references/queueing-theory-applied.md — Load when sizing partitions, modeling lag SLOs, or scaling coordinator throughput.
• references/streaming-patterns.md — Load for starter code: Kafka producer/consumer, Kafka Connect, Flink and Spark streaming jobs, and Kappa/Lambda sketches.
• assets/template-kafka-ingestion.md — Load when scaffolding Kafka ingestion into a lake: producers, batch consumers, connectors, and lag alerts.
• references/kafka-upgrades-and-client-defaults.md — Load when planning a Kafka broker upgrade or ZooKeeper-to-KRaft migration, adopting KIP-848 or share groups, or checking producer defaults for a specific client.
• references/distributed-systems-applied.md — Load when reasoning about ISR quorum, exactly-once via idempotency, leader-epoch fencing, or consumer-group rebalance correctness.
• Formal methods — Load only when a replay, offset/side-effect, rebalance, or fencing protocol has an explicit invariant to check. Return the state/transition boundary and counterexample or bounded safety result; keep delivery guarantees conditional on runtime semantics. Skip routine connector setup.
• data/sources.json — Primary-source URLs for all platforms, processors, CDC tools, and table formats.

Current-Source Policy

• Prefer trust_tier: primary entries in data/sources.json for platform capabilities, service limits, compatibility, and release-sensitive behavior.
• For recommendation questions, verify current managed-service behavior, connector support, quotas, and pricing from official docs instead of relying on frozen comparisons.
• Separate verified facts from judgment calls when comparing Kafka, Redpanda, Pulsar, Kinesis, Flink, and managed offerings.
• If web access is unavailable, say the recommendation is partially unverified.

Related Skills

• data-lake-platform for storage formats, catalogs, serving layers, and lakehouse architecture
• data-analytics-engineering for marts, semantic layers, and metric governance on top of streaming data
• data-sql-optimization for database-side performance and transactional operations
• software-architecture-design for application eventing, CQRS, and domain architecture
• ops-devops-platform for deployment, infra automation, and runbook operations

Learnings Loop

When prior decisions or pitfalls are relevant, consult learnings.consolidated.md if present; use learnings.md only for needed history or as the available fallback. Otherwise skip both.

After applying it, if you encountered a pattern worth remembering, a mistake worth preventing, or a domain fact that surprised you, append one dated bullet to learnings.md via agents-skills-feedback-loop/scripts/append_learning.py. Do not modify SKILL.md itself.