「data」の検索結果

数据工程 — 数据平台从业者的认知操作系统, 覆盖把数据从源系统搬运成可靠 / 可查询 / 可信赖形态供分析 / ML / 数据产品消费的全生命周期 (生成 → 摄取 → 存储 → 转换 → 服务 + 安全/数据管理/DataOps/数据架构/编排/软件工程 六条暗流, Reis & Housley 框架): 摄取与集成 (批 + CDC 变更数据捕获 Debezium + EL 工具 Fivetran/Airbyte/Meltano/dlt + Kafka Connect + schema drift) / 存储与文件表格式 (对象存储数据湖 + 列存 Parquet/ORC/Arrow/Avro + 开放表格式 Apache Iceberg/Delta Lake/Apache Hudi + lakehouse + 分区/compaction) / 转换与建模 (ELT dbt/SQLMesh + Spark + 维度建模 Kimball + Inmon + Data Vault + 大宽表 OBT + 渐变维 SCD + 增量模型 + 语义/指标层) / 编排与工作流 (Apache Airflow/Dagster/Prefect/Mage/Kestra/Apache DolphinScheduler + DAG + 幂等 + 回填 backfill + 数据资产调度) / 批流与实时 (Apache Kafka/Apache Flink/Spark Structured Streaming/Kinesis/Pulsar/Redpanda + Lambda vs Kappa + watermark/窗口/exactly-once + 流式 SQL Materialize/RisingWave + 实时 OLAP ClickHouse/Apache Druid/Apache Pinot/StarRocks/Apache Doris) / 数仓与查询引擎 (Snowflake/BigQuery/Redshift/Databricks SQL/Trino/Presto/DuckDB/Polars + 存算分离 + MPP) / 数据质量测试与可观测性 (dbt tests/Great Expectations/Soda + 数据契约 + Monte Carlo data downtime + 新鲜度/量/schema 异常检测) / 数据治理编目与血缘 (DataHub/Amundsen/OpenMetadata/Unity Catalog + 列级血缘 + PII 分类 + 访问控制 + GDPR) / DataOps 与可靠性 (数据 CI/CD + 转换版本控制 + 环境隔离 + 幂等重处理 + 数据 SLA/SLO + 计算存储 FinOps) / 数据架构范式 (现代数据栈 + lakehouse + data mesh + data fabric + 去中心化 vs 中心化所有权) / 分析工程角色 (dbt 时代连接数据工程与分析的桥) — 不含 数据科学/ML 建模本身 (是下游消费者) / BI 仪表盘制作 (serving 下游) / 数据分析报表为终点 / 'data engineer = 跑 Hadoop 的' 过时窄化 / 通用后端应用开发 (平行学科) (Data Engineering — the cognitive operating system of practitioners who design, build, and operate the data platform: moving data from source systems into reliable, queryable, trustworthy form for analytics / ML / products, covering (a) the data engineering lifecycle (generation → ingestion → storage → transformation → serving, with the undercurrents security / data management / DataOps / data architecture / orchestration / software engineering — Reis & Housley framing), (b) ingestion & integration (batch + CDC change-data-capture with Debezium, EL tools Fivetran / Airbyte / Meltano / dlt, Kafka Connect, API + file + database sources, schema drift handling), (c) storage & file/table formats (object storage data lakes, columnar formats Parquet / ORC / Arrow / Avro, open table formats Apache Iceberg / Delta Lake / Apache Hudi, lakehouse architecture, partitioning / compaction / Z-ordering), (d) transformation & modeling (ELT with dbt / SQLMesh, Spark, dimensional modeling Kimball, Inmon CIF, Data Vault, One Big Table / wide tables, normalization vs denormalization, slowly changing dimensions, incremental models, the semantic / metrics layer), (e) orchestration & workflow (Apache Airflow, Dagster, Prefect, Mage, Kestra, Apache DolphinScheduler, DAGs, idempotency, backfills, data-aware / asset-based scheduling), (f) batch vs streaming & real-time (Apache Kafka, Apache Flink, Spark Structured Streaming, Kinesis / Pulsar / Redpanda, the Lambda vs Kappa debate, watermarks / windowing / exactly-once, streaming SQL Materialize / RisingWave, real-time OLAP ClickHouse / Apache Druid / Apache Pinot / StarRocks / Apache Doris), (g) warehouses & query engines (Snowflake, BigQuery, Redshift, Databricks SQL, Trino / Presto, DuckDB, Polars, decoupled storage & compute, MPP), (h) data quality, testing & observability (dbt tests, Great Expectations, Soda, data contracts, Monte Carlo / data downtime, freshness / volume / schema anomaly detection, unit / integration testing of pipelines), (i) data governance, catalog & lineage (DataHub, Amundsen, OpenMetadata, Unity Catalog, column-level lineage, PII / data classification, access control, GDPR / data privacy), (j) DataOps & reliability (CI/CD for data, version control of transformations, environments, idempotent reprocessing, SLAs / SLOs for data, cost / FinOps for compute & storage), (k) data architecture paradigms (modern data stack, data lakehouse, data mesh, data fabric, decentralized vs centralized ownership), (l) the analytics engineering role (the dbt-era bridge between data engineering and analysis); N

日本語の概要は準備中です。原文の説明を表示しています。

Chief Data Officer advisory for startups: AI training data rights and consent provenance, data product strategy (warehouse vs lakehouse vs mesh, build-vs-buy), B2B customer-data-as-asset valuation and M&A readiness, data team org evolution. Use when deciding whether to train models on customer data, choosing data architecture, valuing data for fundraising or M&A, sequencing data hires, or when user mentions CDO, chief data officer, data strategy, data mesh, lakehouse, training data, data product, data monetization, or customer data asset. NOT a tactical data engineering skill — strategic decisions only.

日本語の概要は準備中です。原文の説明を表示しています。

Use when enabling Data Cloud (Data 360) in an org, creating data spaces, assigning permission sets, registering ingestion API sources, or configuring activation targets. Triggers: 'enable data cloud', 'data spaces setup', 'data cloud permission sets', 'ingestion API connected app', 'activation target setup', 'data cloud licensing', 'dedicated home org vs existing org'. NOT for data streams or DLO-to-DMO mapping — use data/data-cloud-data-streams. NOT for identity resolution rulesets — use admin/data-cloud-identity-resolution.

日本語の概要は準備中です。原文の説明を表示しています。

Use when creating or managing CRM Analytics datasets: configuring dataflows, scheduling refreshes, selecting fields, handling date types, managing row-count growth, or troubleshooting stale or broken datasets. Triggers: 'dataset creation', 'dataflow schedule', 'dataflow quota', 'date field not filterable', 'dataset row limit', 'CRM Analytics data refresh', 'field type mismatch in dataset'. NOT for making an existing dataset smaller or faster — use data/analytics-dataset-optimization. NOT for choosing between Reports, CRM Analytics and Tableau in the first place — use admin/einstein-analytics-basics.

日本語の概要は準備中です。原文の説明を表示しています。

Use this skill when configuring Data Manager in CRM Analytics: enabling objects for sync, scheduling data sync runs, setting up remote connections to external databases, monitoring sync status and error logs, or troubleshooting connected object issues. Trigger keywords: data sync, connected objects, Data Manager, sync schedule, remote connection, Snowflake connector, sync error, incremental sync, CRM Analytics data pipeline. NOT for pushing external data in programmatically via the External Data API — use data/analytics-external-data. NOT for authoring dataflow nodes — use admin/analytics-dataflow-development.

日本語の概要は準備中です。原文の説明を表示しています。

Find and assess datasets for a research question. Dispatches Explorer agents to search across data source categories, then Explorer-Critic to stress-test each candidate. Produces a ranked list with feasibility grades. Make sure to use this skill whenever the user wants to identify or evaluate data sources — not to search for papers or run analysis. Triggers include: "find data", "what data should I use", "find a dataset for this", "where can I get data on X", "assess datasets", "what datasets exist for", "help me find data", "is there data on this", "what are my data options", "I need data for this project", or any request to locate empirical data sources for a research question.

日本語の概要は準備中です。原文の説明を表示しています。

Reads from, writes to, and manages relational databases using MATLAB Database Toolbox. Use when connecting to databases, reading data with sqlread or fetch, filtering with rowfilter, writing with sqlwrite, updating with sqlupdate, executing SQL statements, managing transactions with commit and rollback, mapping MATLAB classes to tables with ORM (Mappable, ormread, ormwrite, ormupdate), or performing any database operation from MATLAB. Triggers on: database, SQL, sqlread, sqlwrite, sqlupdate, fetch, execute, rowfilter, RowFilter, ORM, Mappable, ormread, ormwrite, ormupdate, orm2sql, transaction, commit, rollback, Database Toolbox, PostgreSQL, MySQL, SQLite, SQL Server, Oracle, database connection, database table, query database, insert data, update rows, delete rows, stored procedure, prepared statement, odbc, databaseConnectionOptions, datasource, data source, DSN, connection string, multithreaded, parallel.

日本語の概要は準備中です。原文の説明を表示しています。

Connect MATLAB to Databricks via Spark (Databricks Connect) or JDBC (Database Toolbox). Use when setting up the MATLAB Interface for Databricks, configuring authentication (OauthU2M, OauthM2M, PAT), creating Spark sessions with getDatabricksSession(), reading Unity Catalog tables with server-side filtering, creating JDBC connections with databricks.JDBCConnection or StandaloneJDBCConnection, connecting to SQL Warehouses, selecting JDBC drivers (Simba/OSS), or writing data back to Databricks. Triggers on: Databricks Connect, Spark from MATLAB, getDatabricksSession, .databrickscfg, databricks.JDBCConnection, StandaloneJDBCConnection, SQLWarehouse, Databricks JDBC, Databricks cluster, large table server-side filtering.

日本語の概要は準備中です。原文の説明を表示しています。

Retrieves, versions, and publishes scientific datasets with DataLad and git-annex, and captures computational provenance with datalad run, rerun, and containers-run. Use when cloning or fetching data from OpenNeuro, DANDI, datasets.datalad.org, or any DataLad dataset; when a file in a dataset reads as a broken symlink or a small pointer instead of real data; when an analysis needs a machine-readable record of how each output was produced so it can be re-executed; or when publishing a dataset to siblings such as a GitHub repository plus a storage remote. Also use to decide between DataLad and plain Git for a data-carrying repository.

日本語の概要は準備中です。原文の説明を表示しています。

Prepare, audit, or revise Nature-ready Data Availability statements, data repository plans, dataset citations, and FAIR metadata checklists for manuscripts. Use when the user asks about Nature data availability, research data sharing, repository selection, accession numbers, restricted or sensitive data, source data, supplementary datasets, DataCite-style dataset references, FAIR metadata for academic publication, or Chinese-to-English data availability wording for Chinese-speaking authors preparing Nature-family submissions.

日本語の概要は準備中です。原文の説明を表示しています。

Create original surveys, benchmarks, and aggregated data nobody else has. Automate data collection for content moats. Triggers on: "create original data", "proprietary data", "survey design", "benchmark study", "original research", "data-driven content", "create a survey", "industry benchmark", "aggregated data", "unique data", "first-party data", "data moat", "generate research data", "create a study", "original statistics", "data nobody else has", "competitive data advantage".

日本語の概要は準備中です。原文の説明を表示しています。

Use when you need to apply data-oriented programming best practices in Java — including separating code (behavior) from data structures using records, designing immutable data with pure transformation functions, keeping data flat and denormalized with ID-based references, starting with generic data structures converting to specific types when needed, ensuring data integrity through pure validation functions, and creating flexible generic data access layers. This should trigger for requests such as Improve the code with Data-Oriented Programming; Apply Data-Oriented Programming; Refactor the code with Data-Oriented Programming; Model Java data with records and pure functions; Separate Java behavior from immutable data structures; Validate data integrity with pure Java functions. Part of Plinth Toolkit

日本語の概要は準備中です。原文の説明を表示しています。

Create and manage test data with factory patterns, fixture strategies, data anonymization, and synthetic data generation. Covers Fishery (TypeScript), FactoryBot (Ruby), Factory Boy (Python), database seeding, cleanup strategies, and GDPR-compliant data handling. Use when: "test data," "fixtures," "factories," "seed data," "synthetic data," "test database," "data anonymization." Not for: migration/integrity testing of the DB itself — use database-testing; environment provisioning and database branching strategy — use test-environments. Related: test-environments, database-testing, api-testing, unit-testing.

日本語の概要は準備中です。原文の説明を表示しています。

Complete guide for AutoFixture and xUnit integration. Use when you need to use AutoData or InlineAutoData to simplify xUnit parameterized test data preparation. Covers custom Customization and test data attributes to improve test readability and maintainability. Keywords: AutoData, InlineAutoData, AutoFixture xUnit, [AutoData], [InlineAutoData], AutoDataAttribute, ICustomization, DataAttribute, parameterized test, Theory AutoData, MemberAutoData, test data attributes, fixture.Customize

日本語の概要は準備中です。原文の説明を表示しています。

Data engineering skill for building scalable data pipelines, ETL/ELT systems, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, and modern data stack. Includes data modeling, pipeline orchestration, data quality, and DataOps. Use when designing data architectures, building data pipelines, optimizing data workflows, implementing data governance, or troubleshooting data issues.

日本語の概要は準備中です。原文の説明を表示しています。

Import data into the AWS data lake from S3 files, local uploads, JDBC databases (Oracle, SQL Server, PostgreSQL, MySQL, RDS, Aurora), Amazon Redshift, Snowflake, BigQuery, DynamoDB, or existing Glue catalog tables (migration). Default target is S3 Tables; standard Iceberg on a general purpose bucket is supported where S3 Tables is not adopted. Handles one-time loads, recurring pipelines, migrations. Triggers on: import data, load data, ingest, sync database, migrate table, move data to AWS, set up pipeline, ETL, pull from Snowflake, query BigQuery into S3, export DynamoDB, CTAS, convert to Iceberg. Do NOT use for setting up or troubleshooting Glue connections (use connecting-to-data-source), creating empty tables (use creating-data-lake-table), running queries (use querying-data-lake), finding tables by fuzzy name (use finding-data-lake-assets), catalog audit (use exploring-data-catalog), or SaaS platforms like Salesforce, ServiceNow, SAP, MongoDB, Kafka.

日本語の概要は準備中です。原文の説明を表示しています。

World-class data engineering skill for building scalable data pipelines, ETL/ELT systems, real-time streaming, and data infrastructure. Expertise in Python, SQL, Spark, Airflow, dbt, Kafka, Flink, Kinesis, and modern data stack. Includes data modeling, pipeline orchestration, data quality, streaming quality monitoring, and DataOps. Use when designing data architectures, building batch or streaming data pipelines, optimizing data workflows, or implementing data governance.

日本語の概要は準備中です。原文の説明を表示しています。

Expert knowledge for Azure Data Share development including troubleshooting, decision making, security, configuration, and deployment. Use when estimating Data Share costs, securing invitations, fixing share errors, configuring datasets, or deploying across regions, and other Azure Data Share related development tasks. Not for Azure Data Box (use azure-data-box-family), Azure Data Explorer (use azure-data-explorer), Azure Data Factory (use azure-data-factory), Azure Open Datasets (use azure-open-datasets).

日本語の概要は準備中です。原文の説明を表示しています。

Pre-breach impact analysis: inventories sensitive data (PII, PHI, PCI-DSS, credentials), traces data flows, scores exposure vectors, and produces a regulatory blast radius report with fine ranges sourced verbatim from GDPR Art. 83, CCPA § 1798.155(a), and HIPAA 45 CFR § 160.404. Cost benchmarks from IBM Cost of a Data Breach Report (annually updated). All citations in references/SOURCES.md for verification. Use when asked: "assess breach impact", "what data could be exposed", "calculate blast radius", "data exposure analysis", "how bad would a breach be", "quantify data risk", "sensitive data inventory", "data flow security audit", "pre-breach assessment", "worst-case breach scenario", "breach readiness", "data risk report", "/data-breach-blast-radius". For any stack handling user data, health records, or financial information. Output labels law-sourced figures (exact) vs heuristic estimates (planning only). Does not replace legal counsel.

日本語の概要は準備中です。原文の説明を表示しています。

Build custom Python data sources for Apache Spark using the PySpark DataSource API — batch and streaming readers/writers for external systems. Use this skill whenever someone wants to connect Spark to an external system (database, API, message queue, custom protocol), build a Spark connector or plugin in Python, implement a DataSourceReader or DataSourceWriter, pull data from or push data to a system via Spark, or work with the PySpark DataSource API in any way. Even if they just say "read from X in Spark" or "write DataFrame to Y" and there's no native connector, this skill applies.

日本語の概要は準備中です。原文の説明を表示しています。

Design the UX of custom-code Databricks Apps (AppKit/React) data screens — KPI/overview pages, reports, charts, tables, and Genie/chat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit/React app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading/empty/error), IBCS notation, and AI-result trust (showing generated SQL/sources for Genie/chat). A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding/build/deploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie/chat/AI surface.

日本語の概要は準備中です。原文の説明を表示しています。

Guides compliance with India's Digital Personal Data Protection Act 2023. Covers consent manager registration, data fiduciary obligations under Sections 4-7, significant data fiduciary requirements under Section 10, data principal rights, and Board enforcement framework. Keywords: DPDP Act, India data protection, consent manager, data fiduciary, significant data fiduciary, data principal rights.

日本語の概要は準備中です。原文の説明を表示しています。

Data pipeline specialist for ETL design, data quality, CDC patterns, and batch/stream processingUse when "data pipeline, etl, cdc, data quality, batch processing, stream processing, data transformation, data warehouse, data lake, data validation, data-engineering, etl, cdc, batch, streaming, data-quality, dbt, airflow, dagster, data-pipeline, ml-memory" mentioned.

日本語の概要は準備中です。原文の説明を表示しています。

When the user needs to migrate data between databases, transform schemas, or consolidate data sources. Use when the user mentions "data migration," "database migration," "migrate from MySQL to PostgreSQL," "schema migration," "ETL pipeline," "data transfer," "database consolidation," "legacy migration," or "move data between databases." Covers schema analysis, mapping, transformation, batch processing, validation, and cutover planning. For query optimization during migration, see sql-optimizer.

日本語の概要は準備中です。原文の説明を表示しています。