本文へ移動
cccskills
無料GitHub で公開

data-engineering

Builds and operates data pipelines — ingestion, transformation, orchestration, quality testing, and reliability of data delivery. Use this to design or debug a pipeline, decide batch versus streaming, add data quality checks, handle late or duplicate data, or work out why a dashboard's numbers changed without anyone changing the dashboard.

インストール方法を見る

含まれるファイル(2)

  • SKILL.md3.7 KB
  • references/sources.md2.3 KB

SKILL.md(原文)

インストールする前に、エージェントに与えられる指示の中身を確認できます。

Data engineering

Pipelines are production systems whose failures are quiet. A broken service pages someone; a broken pipeline produces plausible numbers that people act on for a week.

This is movement and transformation. Schema and semantics belong to data-analytics:data-modeling, policy and stewardship to data-analytics:data-governance.

Land raw, transform downstream

Keep an immutable copy of source data exactly as received. Transformation logic will be wrong at some point, and raw data is what lets you reprocess rather than re-request from a source that may no longer have it.

Business logic belongs downstream where it is visible and testable, not buried in ingestion. The exception is transformation required for privacy — minimization, pseudonymization, dropping fields you have no basis to hold — which belongs at ingest precisely because raw storage is what the obligation attaches to. See legal-risk:privacy-and-data-protection.

Idempotence is the property that matters

Every pipeline will be re-run: after a failure, after a fix, after a late-arriving correction. A re-run that double-counts is worse than a failure, because it produces a wrong answer silently.

Design for exactly-once effect at the destination — deterministic keys, merges rather than blind appends, partitioned overwrites. Then re-running is safe and recovery stops being frightening.

Late, duplicate and out-of-order data

Real sources deliver all three. Decide explicitly, per pipeline: how late is an event still accepted, what happens to one arriving after its window closed, and how duplicates are identified.

Distinguish event time from processing time and partition on event time. Aggregations built on arrival time silently reassign yesterday's activity to today whenever a delivery is delayed.

Test data, not just code

Unit tests on transformation logic catch the wrong class of failure. Most damage comes from data that is valid but wrong. Assert on the data itself, in the pipeline, and fail loudly:

  • Row counts within an expected range, not merely non-zero.
  • Uniqueness of keys, and referential integrity across joins.
  • Freshness — the newest record is recent enough to be meaningful.
  • Distribution shifts in important columns.

A silent failure is worse than a loud one. Prefer stopping the pipeline to publishing data you do not trust.

Sources

references/sources.md in this skill lists the outside authorities that settle the questions here — what each one is authoritative for, and what you may do with it. Check them before answering on anything they cover, and cite what you used. Most are free to read and not free to reproduce; the use note on each is binding.

Tooling

Warehouses and lakehouses: Snowflake, BigQuery, Databricks, Redshift, and Postgres or DuckDB at small scale, and similar.

Ingestion: Fivetran, Airbyte, Stitch, and similar. Transformation: dbt, SQLMesh. Orchestration: Airflow, Dagster, Prefect, and similar.

Buy ingestion and build transformation. Connector maintenance returns nothing for the time your team puts into it.

Never

  • Transform on ingest for business reasons and discard the raw copy.
  • Build a pipeline whose re-run double-counts.
  • Aggregate on processing time when event time is available.
  • Let a pipeline fail silently and publish stale data as current.

レビュー

まだレビューはありません。使ってみた感想をお寄せください。

同じリポジトリのスキル

概要と使いどころ

Designs and audits who can reach what — authentication, authorization models, privileged access, service credentials, and joiner-mover-leaver process. Use this to design a permissions model, run an access review, reduce standing privilege, handle offboarding, set up SSO or MFA, manage service and machine credentials, or diagnose why permissions have sprawled.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

Concentrates marketing and sales effort on a named set of accounts rather than on volume — qualifying whether the model fits your economics at all, building the account list and the buying group inside each, tiering effort against account value, coordinating so the account experiences one campaign rather than several, and measuring account progression instead of leads. Use this to decide whether to run an account-based program, build one, or work out why an existing one produces activity and no pipeline.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

Gets new users from signup to first real value — signup flow, onboarding, time-to-value, and the early experience that determines whether someone becomes a user or a lapsed account. Use this to design or fix signup and onboarding, diagnose why signups do not convert to active use, reduce time-to-value, or decide what a new user must accomplish first.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

Designs orchestrator-and-subagent hierarchies for a repository — splitting agents by exclusive write surface, pairing every producer with an independent auditor, and enforcing the split with a script that runs in CI. Use this whenever the user wants to set up, expand, audit, or fix a multi-agent or subagent structure for a codebase; asks how to divide work between agents; wants agent charters, roles, or a surface map written; or is hitting agents that collide on the same files, review their own work, or drift from their remit. Also use when sizing a roster or deciding whether a new agent is justified.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

Governs models and AI systems in production — intended use, evaluation, monitoring, human oversight, documentation, and the decision to deploy or retire. Use this before deploying a model or AI feature, when defining evaluation criteria, when a model's behavior has drifted, when assessing AI risk or regulatory exposure, or when deciding whether an AI system is fit for a consequential decision.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

Produces executive-level research — market sizing, competitor mapping, trend analysis, and strategic intelligence — grounded in cited sources with the confidence in each claim made explicit. Use this to analyze a market or industry, map competitors, evaluate a market-entry or build-versus-buy decision, produce a research brief, or assemble evidence for a decision. Also use when comparing options that need a structured, evidence-based verdict rather than an opinion.

日本語の概要は準備中です。原文の説明を表示しています。

cbrock84/headcount2,0282026年9月18日 更新

cbrock84 のスキルをすべて見る

このスキルの問題を報告する