Create Agent Bricks: Knowledge Assistants (KA) for document Q&A and Supervisor Agents for multi-agent orchestration (MAS).
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新
無料GitHub で公開
databricks-spark-structured-streaming
Comprehensive guide to Spark Structured Streaming for production workloads. Use when building streaming pipelines, working with Kafka ingestion, implementing Real-Time Mode (RTM), configuring triggers (processingTime, availableNow), handling stateful operations with watermarks, optimizing checkpoints, performing stream-stream or stream-static joins, writing to multiple sinks, or tuning streaming cost and performance.
含まれるファイル（15）
- SKILL.md3.7 KB
- agents/openai.yaml414 B
- assets/databricks.png15.0 KB
- assets/databricks.svg582 B
- references/checkpoint-best-practices.md8.4 KB
- references/kafka-streaming.md16.0 KB
- references/lakebase-sink-python.md20.6 KB
- references/merge-operations.md10.7 KB
- references/multi-sink-writes.md12.6 KB
- references/real-time-mode.md18.8 KB
- references/stateful-operations.md11.7 KB
- references/stream-static-joins.md13.9 KB
- references/stream-stream-joins.md16.5 KB
- references/streaming-best-practices.md9.4 KB
- references/trigger-and-cost-optimization.md13.6 KB
SKILL.md（原文）
インストールする前に、エージェントに与えられる指示の中身を確認できます。
Spark Structured Streaming
Production-ready streaming pipelines with Spark Structured Streaming. This skill provides navigation to detailed patterns and best practices.
Quick Start
from pyspark.sql.functions import col, from_json
# Basic Kafka to Delta streaming
df = (spark
.readStream
.format("kafka")
.option("kafka.bootstrap.servers", "broker:9092")
.option("subscribe", "topic")
.load()
.select(from_json(col("value").cast("string"), schema).alias("data"))
.select("data.*")
)
df.writeStream \
.format("delta") \
.outputMode("append") \
.option("checkpointLocation", "/Volumes/catalog/checkpoints/stream") \
.trigger(processingTime="30 seconds") \
.start("/delta/target_table")
Core Patterns
| Pattern | Description | Reference |
|---|---|---|
| Kafka Streaming | Kafka to Delta, Kafka to Kafka, Real-Time Mode | See references/kafka-streaming.md |
| Real-Time Mode (RTM) | Sub-second E2E latency — cluster setup, slot math, supported ops (incl. stream-stream inner join on DBR 18+), transformWithState, observability, error classes, delivery semantics | See references/real-time-mode.md |
| Lakebase Sink | Write streaming records into Lakebase Postgres with transactional upserts. Native format("postgresql") sink (DBR 18.3+) and manual foreach sink as a fallback | See references/lakebase-sink-python.md |
| Stream Joins | Stream-stream joins, stream-static joins | See references/stream-stream-joins.md, references/stream-static-joins.md |
| Multi-Sink Writes | Write to multiple tables, parallel merges | See references/multi-sink-writes.md |
| Merge Operations | MERGE performance, parallel merges, optimizations | See references/merge-operations.md |
Configuration
| Topic | Description | Reference |
|---|---|---|
| Checkpoints | Checkpoint management and best practices | See references/checkpoint-best-practices.md |
| Stateful Operations | Watermarks, state stores, RocksDB configuration | See references/stateful-operations.md |
| Trigger & Cost | Trigger selection, cost optimization, RTM | See references/trigger-and-cost-optimization.md |
Best Practices
| Topic | Description | Reference |
|---|---|---|
| Production Checklist | Comprehensive best practices | See references/streaming-best-practices.md |
Production Checklist
- Checkpoint location is persistent (UC volumes, not DBFS)
- Unique checkpoint per stream
- Fixed-size cluster (no autoscaling for streaming)
- Monitoring configured (input rate, lag, batch duration)
- Exactly-once verified (txnVersion/txnAppId)
- Watermark configured for stateful operations
- Left joins for stream-static (not inner)
レビュー
まだレビューはありません。使ってみた感想をお寄せください。
同じリポジトリのスキル
概要と使いどころ
Use Databricks built-in AI Functions (ai_classify, ai_extract, ai_summarize, ai_mask, ai_translate, ai_fix_grammar, ai_gen, ai_analyze_sentiment, ai_similarity, ai_parse_document, ai_prep_search, ai_query, ai_forecast) to add AI capabilities directly to SQL and PySpark pipelines without managing model endpoints. Also covers document parsing and building custom RAG pipelines (parse → prep_search → index → query).
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新
Databricks AI Runtime, the `databricks air` CLI commands for submitting and managing GPU training workloads on Databricks serverless compute. Use for: writing and submitting `databricks air` workload YAML, passing hyperparameters and secrets, checking run status, listing/cancelling runs, streaming a run's logs and watching its progress, custom Docker image setup, and environment configuration.
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新
Create Databricks AI/BI dashboards. Must use when creating, updating, or deploying Lakeview dashboards as Databricks Dashboard have a unique json structure. CRITICAL: You MUST test ALL SQL queries via CLI BEFORE deploying. Follow guidelines strictly.
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新
Design the UX of custom-code Databricks Apps (AppKit/React) data screens — KPI/overview pages, reports, charts, tables, and Genie/chat data assistants — mapped to concrete AppKit components. Use when BUILDING or reviewing the UI of an AppKit/React app that displays data or answers data questions: choosing genre, layout, charts, KPIs, semantic color, required states (loading/empty/error), IBCS notation, and AI-result trust (showing generated SQL/sources for Genie/chat). A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, NOT this skill. Also NOT for non-data frontend (forms, settings, auth, marketing) or scaffolding/build/deploy (→ databricks-apps). Complements databricks-apps; use it alongside whenever a custom app has a chart, table, KPI, report, or Genie/chat/AI surface.
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新
Build apps on Databricks Apps platform. Use when asked to create data apps, analytics tools, or custom interactive visualizations. A plain "create a dashboard" request means a managed AI/BI (Lakeview) dashboard → use databricks-aibi-dashboards, not this skill. Evaluates data access patterns (analytics vs Lakebase synced tables) before scaffolding. Invoke BEFORE starting implementation.
日本語の概要は準備中です。原文の説明を表示しています。
databricks/databricks-agent-skills3452026年10月10日 更新